Image detection method and storage medium

Abstract

The present application provides an image detection method. The method comprises: acquiring a query image and one or more prompt information pairs, wherein each prompt information pair comprises a prompt image and a prompt truth value corresponding to the prompt image, the one or more prompt information pairs are used for prompting the type of a first detection task, and the prompt truth value is used for representing a result obtained after the first detection task is executed on the corresponding prompt image; and inputting the query image and the one or more prompt information pairs into an image detection model, and executing the first detection task on the query image by means of the image detection model and on the basis of the prompt image and the corresponding prompt truth value to obtain a detection result, wherein the image detection model has the function of executing different types of detection tasks on the basis of different prompt information pairs, and different types of detection tasks comprise the first detection task.

Description

Image detection method and storage medium Technical Field

[0001] The present application relates to the field of deep learning, and in particular to an image detection method, apparatus, device and storage medium. Background Art

[0002] In the field of industrial inspection, due to the wide variety of industrial products and the numerous types of anomalies within each category, manual inspection is inefficient and its accuracy is easily affected by the inspector's individual experience and work status. To improve inspection efficiency and accuracy, related technologies have proposed using deep learning models to inspect industrial product images. However, these deep learning models can only detect a single type of anomaly within a fixed set of industrial products and are unable to handle the diverse range of industrial products. Summary of the Invention

[0003] Based on the above technical problems, the present application provides an image detection method, an image detection device, an electronic device, a computer program product and a readable storage medium, which can prompt a detection model with the function of performing different detection tasks to perform different detection tasks through prompt information, and is compatible with a wide variety of industrial products.

[0004] In a first aspect, the present application provides an image detection method, which includes: obtaining a query image and one or more prompt information pairs, each prompt information pair including a prompt image and a prompt truth value corresponding to the prompt image; one or more prompt information pairs are used to indicate the type of a first detection task; the prompt truth value is used to represent the result obtained after performing the first detection task on the corresponding prompt image; the query image and one or more prompt information pairs are input into an image detection model, and the image detection model performs the first detection task on the query image based on the prompt image and the corresponding prompt truth value to obtain a detection result; the image detection model has the function of performing different types of detection tasks according to different prompt information pairs; different types of detection tasks include the first detection task.

[0005] Optionally, the image detection model includes a feature encoding network, an information aggregation network, and a decoding network; the image detection model performs a first detection task on the query image to obtain a detection result, including: extracting a query feature map of the query image and a prompt feature map of one or more prompt images through the feature encoding network; the size of the query feature map and the prompt feature map is smaller than the size of the query image; the query feature map, the prompt feature maps of one or more prompt images, and the prompt true values ​​corresponding to the one or more prompt images are aggregated through the information aggregation network to obtain an aggregated feature map; the aggregated feature map is decoded and upsampled through the decoding network until it is mapped to the size of the query image to obtain the detection result of the first detection task.

[0006] Optionally, the hint truth value is a mask; after obtaining the query image and one or more hint information pairs, the method further includes: mapping the mask to a d-dimensional task space; d is a positive integer greater than 1; after performing a first detection task on the query image based on the hint image and the corresponding hint truth value through the image detection model and obtaining the detection result, the method further includes: reverse mapping the detection result from the d-dimensional task space to the modality of the mask.

[0007] Optionally, each dimension of the d-dimensional task space corresponds to a channel; the information aggregation network includes multiple cross-attention units, each cross-attention unit includes an attention layer and a feedforward network; the query feature map, the prompt feature maps of one or more prompt images, and the prompt true value corresponding to one or more prompt images are aggregated by the information aggregation network to obtain an aggregated feature map, including: splicing the prompt feature map and the prompt true value corresponding to each prompt image in the channel dimension, and mapping them into the first feature map and the second feature map corresponding to each prompt image through a convolutional network; performing a first operation on the query feature and the first feature map and the second feature map corresponding to a prompt image by each cross-attention unit to obtain a feature map corresponding to a prompt image output by each cross-attention unit; the first operation includes: aggregating the query feature and the first feature map and the second feature map corresponding to a prompt image through the attention layer, and encoding the aggregated feature map through the feedforward network; feature averaging the feature maps corresponding to one or more prompt images output by multiple cross-attention units to obtain an aggregated feature map.

[0008] Optionally, the different types of detection tasks that the image detection model can perform include edge extraction tasks, semantic segmentation tasks, and key point prediction tasks; for the edge extraction task, the mask includes a foreground mask; mapping the mask to a d-dimensional task space includes: setting the elements of each dimension of the d-dimensional task space to 0; replacing the foreground mask with a channel of any dimension in the d-dimensional task space; for the semantic segmentation task, the mask includes targets of different categories; mapping the mask to the d-dimensional task space includes: replacing targets of different categories in the mask with channels of different dimensions in the d-dimensional task space; for the key point prediction task, the mask includes key points of different categories; mapping the mask to the d-dimensional task space includes: replacing key points of different categories in the mask with channels of different dimensions in the d-dimensional task space.

[0009] Optionally, the method also includes: obtaining a training sample set; the training sample set includes training samples corresponding to multiple detection tasks; each training sample includes a training query image, a training query true value corresponding to the training query image, and a training prompt information pair, the training prompt information pair includes a training prompt image, and a training prompt true value corresponding to the training prompt image, the training prompt true value is used to indicate the detection result obtained after performing the training detection task on the corresponding training prompt image; the training detection task is the detection task corresponding to the training sample where the training prompt true value is located; based on the training sample set, the preset model to be trained is trained to obtain an image detection model.

[0010] Optionally, the training query image and the training prompt image in each training sample are the same, and the training query true value and the training prompt true value are the same. Optionally, obtaining a training sample set includes: obtaining a training image, and an original prompt true value and an original query true value of the training image; the original prompt true value and the original query true value are both original masks; each time the following operation is performed: using a local content of the original mask as the training prompt true value and the training query true value corresponding to the training image, to obtain multiple training prompt true values ​​and multiple training query true values ​​corresponding to the training image; each time the following operation is performed: obtaining a training sample corresponding to the training image based on the training image, the training prompt true value corresponding to the training image, and the training query true value corresponding to the training image, to obtain multiple training samples corresponding to the training image; and obtaining a training sample set based on the multiple training samples corresponding to the training image.

[0011] Optionally, obtaining a training sample set includes: obtaining a first training sample; the first training sample includes a first information pair and a second information pair; the first information pair includes a first training query image and a training query true value corresponding to the first training query image; the second information pair includes a first training prompt image and a training prompt true value corresponding to the first training prompt image; performing a transformation operation on the first information pair and / or the second information pair to obtain a second training sample; the transformation operation includes any one or more of the following: rotation, scaling, translation, cropping, flipping, adding noise, and color transformation; and obtaining a training sample set based on the first training sample and the second training sample.

[0012] Optionally, the training prompt true value and the training query true value are training masks, and the training mask includes different types of elements; based on the training sample set, the preset model to be trained is trained to obtain an image detection model, including: randomly mapping elements of each type in the training mask to a channel of any dimension in the d-dimensional task space; inputting the training sample into the model to be trained to obtain an initial prediction result output by the model to be trained; reversely mapping the initial prediction result from the d-dimensional task space to the mode of the training mask to obtain a target prediction result, and completing one training; training the model to be trained multiple times to obtain an image detection model.

[0013] Optionally, the model to be trained includes a feature encoding network, an information aggregation network, and a decoding network; based on the training sample set, the preset model to be trained is trained to obtain an image detection model, including: based on the training sample set, the information aggregation network and the decoding network are trained until the information aggregation network and the decoding network converge; based on the converged information aggregation network and the decoding network, a candidate model is obtained; based on the converged information aggregation network and the decoding network, and the feature encoding network, the candidate model is trained to obtain an image detection model.

[0014] The image detection method provided in the present application can use prompt information pairs to prompt an image detection model that has the function of performing different detection tasks according to different prompt information pairs, so that the image detection model performs the first detection task prompted by the prompt information pair and obtains the detection result obtained by performing the first detection task. Compared with the deep learning model that can only detect one type of abnormality of a fixed type of industrial product, the image detection method provided in the present application can perform different types of detection tasks and is compatible with a wide variety of industrial products and defect types.

[0015] In a second aspect, the present application provides an image detection device, which includes an acquisition module for acquiring a query image and one or more prompt information pairs, each prompt information pair including a prompt image and a prompt true value corresponding to the prompt image; the one or more prompt information pairs are used to indicate the type of a first detection task; the prompt true value is used to represent the result obtained after performing the first detection task on the corresponding prompt image; a processing module for inputting the query image and the one or more prompt information pairs into an image detection model, and performing the first detection task on the query image based on the one or more prompt images and the prompt true value corresponding to each prompt image through the image detection model to obtain a detection result; the image detection model has the function of performing different types of detection tasks according to different prompt information pairs; the different types of detection tasks include the first detection task.

[0016] In a third aspect, the present application provides an electronic device comprising a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method described in the first aspect above.

[0017] In a fourth aspect, the present application provides a computer program product, which, when executed in an electronic device, enables the electronic device to execute the method described in the first aspect above.

[0018] In a fifth aspect, the present application provides a readable storage medium, which includes: software instructions; when the software instructions are executed in an electronic device, the electronic device implements the method described in the first aspect above.

[0019] The beneficial effects of the second to fifth aspects mentioned above can be referred to those described in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] FIG1 is a schematic diagram of the composition of an image detection system provided in an embodiment of the present application.

[0022] FIG2 is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application.

[0023] FIG3 is a flow chart of the image detection method provided in an embodiment of the present application.

[0024] FIG4 is a schematic diagram of an application case provided in an embodiment of the present application.

[0025] FIG5 is another flow chart of the image detection method provided in an embodiment of the present application.

[0026] FIG6 is another flow chart of the image detection method provided in an embodiment of the present application.

[0027] FIG7 is a schematic diagram of the structure of the information aggregation network provided in an embodiment of the present application.

[0028] FIG8 is another structural diagram of the information aggregation network provided in an embodiment of the present application.

[0029] FIG9 is an overall flow chart of the image detection method provided in an embodiment of the present application.

[0030] FIG10 is another flow chart of the image detection method provided in an embodiment of the present application.

[0031] FIG11 is a schematic diagram of the composition of the image detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Thus, a feature designated as "first," "second," or "third," etc., may explicitly or implicitly include one or more of the features.

[0033] Vision-based industrial inspection technology refers to the use of computer vision methods to locate targets in industrial scenarios and perform tasks such as quality inspection. Vision-based industrial inspection technology generally includes the following steps: 1) Image acquisition: Images of industrial products are captured using devices such as high-definition cameras. 2) Image processing and analysis: Computer vision and image processing technologies are used to perform preprocessing, feature extraction, target detection, and recognition on the captured images to achieve automated inspection of the products to be inspected. 3) Data storage and transmission: The processed images are stored in a database and transmitted to the control execution department and other relevant departments to facilitate product quality assessment and fault diagnosis. 4) Control execution department: Based on the results of image analysis, the automated control system is used to control production equipment and adjust production parameters to ensure product quality and production efficiency.

[0034] Vision-based industrial inspection technology, characterized by non-contact, high precision, and high efficiency, has been widely used in various aspects of industry, such as surface defect detection, dimensional measurement, and color recognition. At the same time, with the continuous development of artificial intelligence and computer vision technologies, the level of vision-based industrial inspection technology is also constantly improving, providing strong support for the automation and intelligentization of industrial production.

[0035] The second step in vision-based industrial inspection technology usually uses deep learning models to analyze images of industrial products.

[0036] However, the deep learning model in vision-based industrial inspection technology can only detect one type of abnormality in a fixed type of industrial product and cannot cope with a wide variety of industrial products.

[0037] Based on this, the present invention provides an image detection method that can prompt image detection models capable of performing different types of detection tasks to perform different types of detection tasks through prompt information, and is compatible with a wide variety of industrial products. The following is an introduction with reference to the accompanying drawings.

[0038] Figure 1 is a schematic diagram of the composition of an image detection system provided in an embodiment of the present application. As shown in Figure 1, the system includes: an image acquisition device 100 and an image detection device 200. The image acquisition device 100 and the image detection device 200 can be connected via a wired network or a wireless network.

[0039] The image acquisition device 100 may be an image acquisition device such as a camera or a camera installed in an industrial production scene. For example, the image acquisition device 100 may be a camera installed beside or above a conveyor belt that transports industrial products.

[0040] The image acquisition device 100 can be used to acquire images of industrial products.

[0041] In some embodiments, the image acquisition device 100 may also be configured to send the acquired image to the image detection device 200 .

[0042] As described above, the image acquisition device 100 and the image detection device 200 may be connected via a wired network or a wireless network that may include one or more media or devices capable of transmitting image data from the image acquisition device 100 to the image detection device 200 .

[0043] In some embodiments, the wired or wireless network may include one or more communication media that enable the image acquisition device 100 to transmit image data directly to the image detection device 200 in real time. In this embodiment, the image acquisition device 100 may modulate the image data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated image data to the image detection device 200. The one or more communication media may include wireless and / or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines.

[0044] Alternatively, the one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (such as the Internet).

[0045] Optionally, the one or more communication media may also include a router, a switch, a base station, or other devices that facilitate communication from the image acquisition device 100 to the image detection device 200 .

[0046] The image detection device 200 may be an electronic device with computing and processing capabilities, such as a computer or a server.

[0047] Among them, the server can be a single server, or it can be a server cluster consisting of multiple servers. In some implementations, the server cluster can also be a distributed cluster. Optionally, the server can also be implemented on a cloud platform. For example, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud, and a multi-cloud, or any combination thereof. The embodiments of the present application are not limited to this.

[0048] The image detection device 200 can be used to detect the image from the image acquisition device 100 using a preset image detection model. The specific process can be referred to the image detection method provided in the following embodiment, and will not be repeated here.

[0049] FIG1 above illustrates an example in which the image acquisition device 100 and the image detection device 200 are independent devices. Alternatively, the image acquisition device 100 and the image detection device 200 may be integrated into one device, that is, the image acquisition device 100 or its corresponding functions and the image detection device 200 or its corresponding functions may be integrated into a single device. For example, a camera with image detection functionality may be used. This is not a limitation of the present embodiment.

[0050] The execution subject of the image detection method provided in the embodiment of the present application is the above-mentioned image detection device 200. As mentioned above, the image detection device 200 can be an electronic device with computing and processing functions such as a computer or a server. Optionally, the image detection device 200 can also be a processor (such as a central processing unit (CPU)) in the aforementioned electronic device; or, the image detection device 200 can also be an application (APP) in the aforementioned electronic device; or, the image detection device 200 can also be a software system or platform deployed in the aforementioned electronic device; or, the image detection device 200 can also be a functional module in the aforementioned electronic device for executing the image detection method, etc. The embodiment of the present application does not limit this.

[0051] For simplicity of description, the image detection device 200 is taken as an example of an electronic device for introduction below.

[0052] Figure 2 is a schematic diagram of the components of an electronic device provided in an embodiment of the present application. As shown in Figure 2, the electronic device may include a processor 10, a memory 20, a communication circuit 30, a communication interface 40, and an input / output interface 50. The processor 10, the memory 20, the communication interface 40, and the input / output interface 50 may be connected via the communication circuit 30.

[0053] Processor 10 is used for executing the instructions stored in memory 20 to implement the image detection method provided in the following embodiment of the present application. Processor 10 can be a CPU, a general-purpose processor, a network processor (network processor, NP), a digital signal processor (digital signal processing, DSP), a microprocessor, a microcontroller (micro control unit, MCU), a programmable logic device (programmable logic device, PLD) or any combination thereof. Processor 10 can also be any other device with a processing function, such as a circuit, a device or a software module, and the embodiment of the present application does not limit this. In one example, processor 10 may include one or more CPUs, such as CPU0 and CPU1 in Figure 2. As an optional implementation, the electronic device may include multiple processors, for example, in addition to processor 10, it may also include processor 60 (illustrated as an example by the dotted line in Figure 2).

[0054] Memory 20 is used to store instructions. For example, the instruction can be a computer program. Optionally, the memory 20 can be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, etc., and the embodiments of the present application are not limited to this.

[0055] It should be noted that the memory 20 may exist independently of the processor 10 or may be integrated with the processor 10. The memory 20 may be located inside the electronic device or outside the electronic device, which is not limited in the embodiment of the present application.

[0056] The communication line 30 is used to transmit information between the components included in the electronic device.

[0057] Communication interface 40 is used to communicate with other devices (e.g., image acquisition device 100) or other communication networks. Such other communication networks may include Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. Communication interface 40 may be a module, circuit, transceiver, or any other device capable of communication.

[0058] The input / output interface 50 is used to implement human-computer interaction between a user and the electronic device, for example, to implement action interaction or information interaction between the user and the electronic device.

[0059] For example, the input / output interface 50 may be a mouse, keyboard, display screen, or touch screen screen, etc. Action interaction or information interaction between a user and the electronic device may be achieved through the mouse, keyboard, display screen, or touch screen screen, etc.

[0060] It should be noted that the structure shown in Figure 2 does not constitute a limitation on the electronic device. In addition to the components shown in Figure 2, the electronic device may include more or fewer components than shown in the figure (for example, only a processor 10 and a memory 20), or a combination of certain components, or a different arrangement of components.

[0061] The following is an introduction to the image detection method provided in the embodiments of the present application.

[0062] Figure 3 is a flow chart of the image detection method provided in an embodiment of the present application. Optionally, the method can be performed by an electronic device having the hardware structure shown in Figure 2 above. As shown in Figure 3, the method includes steps S101 to S102.

[0063] Step S101: The electronic device obtains a query image and one or more prompt information pairs.

[0064] The query image can be understood as the image to be inspected. In the field of industrial inspection, the query image can be understood as an image containing an industrial product. Each prompt information pair includes a prompt image and a prompt ground truth value corresponding to the prompt image. The one or more prompt information pairs are used to indicate the type of the first inspection task. The prompt ground truth value represents the result obtained after performing the first inspection task on the corresponding prompt image.

[0065] Exemplarily, the query image and the prompt image in one or more prompt information pairs can be for images of the same industrial product. These one or more prompt information pairs need to correspond to the same detection task, so their scenes are similar and the detection tasks are consistent, such as detecting the same defect on a certain model of product. For example, taking the industrial product as a wooden board as an example, assuming that the user wants to detect scratches in the wooden board, a semantic segmentation task can be performed on the image of the wooden board to segment the normal area and the scratch area. Therefore, in order to prompt the execution of the semantic segmentation task, the prompt information pair can be set to a wooden board image (prompt image) and a mask for representing the position and shape of the scratches in the wooden board image (prompt true value).

[0066] Optionally, as described above, the image acquisition device 100 may transmit the acquired industrial product image to the electronic device (image detection device 200 ). The electronic device (image detection device 200 ) may obtain a query image by receiving the industrial product image from the image acquisition device 100 .

[0067] Optionally, as described above, the electronic device may include an input / output interface 50, which may be a mouse, keyboard, or touch screen, etc. The electronic device may receive the mask input by the user as a prompt truth value through the mouse, keyboard, or touch screen, etc., to obtain a prompt information pair.

[0068] Step S102: The electronic device inputs the query image and one or more prompt information pairs into an image detection model, and the image detection model performs a first detection task on the query image based on the prompt image and the corresponding prompt true value to obtain a detection result.

[0069] The image detection model has the function of performing different detection tasks according to different prompt information, and the different detection tasks include the first detection task. The specific training process of the image detection model can refer to steps S301 to S302 in Figure 10 below, and will not be repeated here. The specific process of step S102 can refer to steps S1021 to S1023 in Figure 6 below, and will not be repeated here.

[0070] For example, the different detection tasks may include: edge extraction tasks, semantic segmentation tasks, key point extraction tasks, etc. The embodiment of the present application does not limit the specific types of detection tasks that can be performed by the image detection model.

[0071] For example, Figure 4 is a schematic diagram of an application case provided by an embodiment of the present application. As shown in Figure 4, taking a wooden board as an example, assuming that the user wishes to detect scratches in the wooden board, the image of the wooden board to be detected (query image), multiple prompt images, and the corresponding prompt true values ​​(task prompts, i.e., the above-mentioned prompt information pairs) can be input into the general industrial detection model (i.e., the above-mentioned image detection model), thereby prompting the general industrial detection model to perform the semantic segmentation task and ultimately segment the query image to obtain the prediction result (a mask representing the position and shape of the scratches in the query image).

[0072] In the image detection method provided in the embodiment of the present application, the electronic device can use prompt information pairs to prompt an image detection model that has the function of performing different detection tasks according to different prompt information pairs, so that the image detection model performs the first detection task prompted by the prompt information pair and obtains the detection result obtained by performing the first detection task. Compared with the deep learning model that can only detect one type of abnormality of a fixed type of industrial product, the image detection method provided in the embodiment of the present application can perform different types of detection tasks and is compatible with a wide variety of industrial products and defect types.

[0073] In some embodiments, in order to make the image detection model compatible with different types of detection tasks, the prompt true value can be uniformly mapped to a multidimensional space unrelated to the detection task before the prompt true value is input into the image detection model, and then the output result of the image detection model is reversely mapped from the d-dimensional task space to the modality of the prompt true value. In this case, taking the prompt true value as a mask as an example, Figure 5 is another flow chart of the image detection method provided in an embodiment of the present application. As shown in Figure 5, after the above-mentioned step S101, the method may further include step S201, and after the above-mentioned step S102, the method may further include step S202.

[0074] Step S201: The electronic device maps the mask to a d-dimensional task space.

[0075] The d-dimensional task space refers to a space with d dimensions that can accommodate different detection tasks. d is a positive integer greater than 1. Each dimension in the d-dimensional task space can correspond to a channel.

[0076] As mentioned above, the different detection tasks that can be performed by the image detection model include edge extraction tasks, semantic segmentation tasks, and key point prediction tasks.

[0077] Optionally, for the edge extraction task, the mask may include a foreground mask, and the above step S201 may specifically include the following steps a1 to a2.

[0078] Step a1: The electronic device sets the elements of each dimension of the d-dimensional task space to 0.

[0079] Step a2: The electronic device replaces the foreground mask to a channel of any dimension in the d-dimensional task space.

[0080] For example, taking the detection of the edge of metal parts as an example, the mask can include a foreground mask and a background mask. The electronic device first initializes the elements of each dimension of the d-dimensional task space to all zeros (that is, all are set to 0), and then the foreground mask including the edge of the metal part can be replaced with the channel of any dimension in the d-dimensional task space.

[0081] Optionally, for the semantic segmentation task, the mask may include targets of different categories, and the above step S201 may specifically include the following step b1: the electronic device replaces targets of different categories in the mask to channels of different dimensions in the d-dimensional task space.

[0082] For example, taking the detection of surface defects of mechanical parts as an example, the mask may include masks of different areas of the mechanical parts, and the electronic device can replace the masks of different areas with channels of different dimensions in the d-dimensional task space.

[0083] Optionally, for the key point prediction task, the mask may include key points of different categories, and the above step S201 may specifically include the following step c1: replacing key points of different categories in the mask to channels of different dimensions in the d-dimensional task space.

[0084] For example, taking the detection of welding points on a circuit board as an example, the mask may include masks of different categories of welding points on the circuit board, and the electronic device may replace the masks of different categories of welding points on channels of different dimensions in the d-dimensional task space.

[0085] Step S202: The electronic device reversely maps the detection result from the d-dimensional task space to the modality of the mask.

[0086] The reverse mapping process can be followed as described above and will not be repeated here. The mask modality is a multidimensional matrix with a shape of H*W*C, where H, W, and C represent the height, width, and dimensions of the matrix. For example, for image segmentation tasks, C is the number of classes, and each point in the matrix is ​​a C-dimensional one-hot code, indicating which class the pixel at that point belongs to.

[0087] The above step S102 is introduced below.

[0088] In some embodiments, FIG6 is another schematic flow chart of an image detection method provided in an embodiment of the present application. As shown in FIG6 , the image detection model includes a feature encoding network, an information aggregation network, and a decoding network. In this case, step S102 may specifically include steps S1021 to S1023.

[0089] Step S1021: The electronic device extracts a query feature map of the query image and respective prompt feature maps of one or more prompt images through a feature encoding network.

[0090] The feature encoding network is used to extract features from the query and prompt images. This network is fundamental to the method's versatility and must be able to extract high-quality features. It can be understood as an encoder, such as the convolutional and fully connected layers of DINOv2 or other convolutional neural networks (CNNs). To improve feature richness, the smaller feature maps output by the last three stages of the feature encoding network can be used. This means that the query and prompt feature maps are smaller than the query image.

[0091] For example, the size of the query feature map and the size of the prompt feature map of one or more prompt images can be the same and both are a first size; the ratio between the first size and the query image size is s, and s is less than 1.

[0092] Exemplarily, s can be 1 / 14 or other values, etc., and the embodiments of the present application are not limited to this.

[0093] Step S1022: The electronic device aggregates the query feature map, the prompt feature maps of the one or more prompt images, and the prompt true values ​​corresponding to the one or more prompt images through the information aggregation network to obtain an aggregated feature map.

[0094] Optionally, Figure 7 is a structural diagram of the information aggregation network provided by an embodiment of the present application. As shown in Figure 7, the information aggregation network may include a plurality of cross-attention units, and each cross-attention unit includes an attention layer and a feedforward network. The attention layer can aggregate the feature maps (Task prompts 1 to task prompts i and the query feature map are shown as examples in Figure 7, and the task prompts are the first feature maps and the second feature maps corresponding to the prompt images below), and the feedforward network can encode the aggregated feature maps and output an encoded feature map. The feature maps output by each cross-attention unit can then be feature averaged to obtain an aggregated feature map.

[0095] Optionally, based on the understanding of the structure shown in FIG. 7 , the step S1022 may specifically include the following steps 1022 - 1 to 1022 - 3 .

[0096] Step 1022-1: The electronic device concatenates the prompt feature map and the prompt true value corresponding to each prompt image in the channel dimension, and maps them into the first feature map and the second feature map corresponding to each prompt image through a convolutional network.

[0097] For example, if the hint feature map includes 1024 dimensions and the hint ground truth also includes 1024 dimensions, the concatenated feature map may include 2048 dimensions. The purpose of mapping the first feature map and the second feature map corresponding to the hint image through the convolutional network includes aligning the dimensions of the query feature. The convolutional network can be designed to output the first feature map and the second feature map.

[0098] Step 1022-2: The electronic device performs a first operation on the query feature and the first feature map and the second feature map corresponding to a prompt image through each cross-attention unit to obtain a feature map corresponding to a prompt image output by each cross-attention unit.

[0099] The first operation includes: aggregating query features, a first feature map and a second feature map corresponding to a prompt image through an attention layer, and encoding the aggregated feature map through a feedforward network.

[0100] Optionally, the attention layer may aggregate the query features, the first feature map, and the second feature map to be aggregated using the following formula (1):

[0101] In formula (1), f Q Represents the query features. Represents the first feature map corresponding to the i-th prompt image. represents the second feature map corresponding to the i-th prompt image. d represents the dimension of the feature map. The input of the attention layer includes the query feature (also called Query), the first feature map (also called Key), and the second feature map (also called Value).

[0102] Optionally, the feature map after feedforward network coding aggregation can be specifically expressed by the following formula (2): Q ←FFN(f Q ) Formula (2)

[0103] Step 1022-3: The electronic device performs feature averaging on the feature maps corresponding to one or more prompt images output by multiple cross-attention units to obtain an aggregated feature map.

[0104] For example, for a feature x, averaging the feature is to take the average value mean(x).

[0105] Optionally, as shown in the structure of the information aggregation network in Figure 7 above, every N (N is a positive integer greater than 1) cross-attention units can be regarded as a group of cross-attention units. In this case, Figure 8 is another structural schematic diagram of the information aggregation network provided in an embodiment of the present application. As shown in Figure 8, the information aggregation network may include multiple aggregation modules, each aggregation module may include a group (N) of cross-attention units, and each aggregation module may be used to aggregate the first feature and the second feature (Task prompt feature is shown as an example in Figure 8) and the query feature obtained by mapping the prompt feature and the prompt true value corresponding to a prompt image to obtain an aggregated feature map. The specific aggregation process can be referred to the description in Figure 7 above, and will not be repeated here.

[0106] Step S1023: The electronic device decodes the aggregated feature map through a decoding network and upsamples it until it is mapped to the size of the query image, thereby obtaining a detection result of the first detection task.

[0107] The decoding network can be understood as a decoder. For example, the regression layer in DINOv2 or other convolutional neural networks (CNNs) can be used as a decoder. The regression layer can be, for example, a softmax regression layer, a sigmoid regression layer, or a regression layer using other regression algorithms. This embodiment of the present application is not limited to this.

[0108] Upsampling refers to interpolating the pixels of the aggregated feature map to increase the number of pixels in the aggregated feature map, thereby increasing the size of the aggregated feature map.

[0109] For example, upsampling may be performed using interpolation methods such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation. The embodiment of the present application does not limit the specific interpolation method used for upsampling.

[0110] Based on the understanding of the above embodiments, FIG9 is an overall flow chart of the image detection method provided by the embodiment of the present application. As shown in FIG9, the electronic device can obtain the query image, the prompt image, and the prompt true value, and perform task space mapping on the prompt true value. Then, the electronic device can input the query image, the prompt image, and the prompt true value into the feature encoding network of the image detection model. The feature encoding network can extract the prompt feature from the prompt image and the query feature from the query image. The information aggregation network can aggregate the mapped prompt true value, the prompt feature, and the query feature, which are then decoded and upsampled by the decoding network to obtain a preliminary prediction result, and the preliminary prediction result is inversely mapped to the task space to obtain the final prediction result. Optionally, the electronic device can also obtain the query true value annotated by the user, perform task space mapping on the query true value, and use the mapped query true value to train the image detection model. The specific training process can refer to the following embodiment and will not be repeated here.

[0111] In some embodiments, before the above step S101, the electronic device may further obtain an image detection model.

[0112] In one possible implementation, the electronic device can directly obtain the trained image detection model from another device. For example, the electronic device can download the trained image detection model from another device or transfer it through an intermediate storage medium.

[0113] In another possible implementation, the electronic device can also use the training samples to train a preset model to be trained, obtaining a trained model as the image detection model. In this case, Figure 10 is another flow chart of the image detection method provided in an embodiment of the present application. As shown in Figure 10, before the above step S101, the method can also include steps S301 to S302.

[0114] Step S301: The electronic device obtains a training sample set.

[0115] The training sample set includes training samples corresponding to multiple detection tasks. Each training sample includes a training query image, a training query true value corresponding to the training query image, and one or more training prompt information pairs. Each training prompt information pair includes a training prompt image and a training prompt true value corresponding to the training prompt image. The training prompt true value is used to indicate the detection result obtained after performing the training detection task on the corresponding training prompt image, and the training detection task is the detection task corresponding to the training sample where the training prompt true value is located. The training query true value can be a manually labeled detection result obtained after performing the training detection task on the training query image. In the field of industrial detection, the training query image can be understood as an image including industrial products. The training detection tasks can include the above-mentioned edge extraction tasks, semantic segmentation tasks, and key point extraction tasks, etc.

[0116] For example, let's say a user wants to train an image detection model to detect scratches on a wooden board. This means they want the model to be able to perform semantic segmentation. In this case, they can use a wooden board image as a training query image, and a mask representing the location and shape of the scratches in the image as the ground-truth training query. They can then use other wooden board images, either identical or different from the original, as training hint images, and use the masks representing the location and shape of the scratches in the other images as the ground-truth training hints to obtain training samples from the training sample set.

[0117] In one possible implementation, the electronic device can perform regional randomization on the ground truth values ​​(training prompt ground truth values ​​and training query ground truth values) in the training samples to obtain multiple different ground truth values, thereby expanding the granularity and diversity of prompts and learning categories and avoiding overfitting of categories. In this case, the above S301 can specifically include the following steps d1 to d4.

[0118] In step d1, the electronic device obtains a training image, and an original hint true value and an original query true value of the training image, wherein the original hint true value and the original query true value are both original masks.

[0119] In step d2, each time the electronic device performs the following operation: using the local content of the original mask as a training hint ground truth value and a training query ground truth value corresponding to a training image, the electronic device obtains multiple training hint ground truth values ​​and multiple training query ground truth values ​​corresponding to the training image. In other words, the electronic device performs this operation multiple times to obtain multiple training hint ground truth values ​​and multiple training query ground truth values ​​corresponding to the training image.

[0120] In step d3, each time the electronic device performs the following operation: based on the training image, the training hint ground truth corresponding to the training image, and the training query ground truth corresponding to the training image, the electronic device obtains a training sample corresponding to the training image, thereby obtaining multiple training samples corresponding to the training image. In other words, the electronic device performs this operation multiple times to obtain multiple training samples corresponding to the training image.

[0121] Step d4: The electronic device obtains a training sample set based on the multiple training samples corresponding to the training image.

[0122] Optionally, the electronic device can obtain multiple training images and expand each training image according to the above steps d1-d4 to obtain multiple training samples corresponding to each training image, and finally form a training sample set together with the multiple training samples corresponding to the training image.

[0123] For example, taking the above-mentioned industrial product as a wooden board as an example, the electronic device can obtain a wooden board image as a training image, a mask representing the position and shape of the scratches in the wooden board image as the original prompt true value and the original query true value, and then use a part of the content of the mask each time as a new training prompt true value and training query true value to obtain multiple training samples corresponding to the wooden board image.

[0124] In another possible implementation, the electronic device may use a random data enhancement strategy to amplify the training sample set. In this case, the above step S301 may specifically include the following steps e1 to e3.

[0125] Step e1: The electronic device obtains a first training sample. The first training sample can be any training sample. The first training sample includes a first information pair and a second information pair. The first information pair includes a first training query image and a training query ground truth corresponding to the first training query image. The second information pair includes a first training prompt image and a training prompt ground truth corresponding to the first training prompt image.

[0126] In step e2, the electronic device performs a transformation operation on the first information pair and / or the second information pair to obtain a second training sample, wherein the transformation operation includes any one or more of the following: rotation, scaling, translation, cropping, flipping, adding noise, and color transformation.

[0127] Step e3: Obtain a training sample set based on the first training sample and the second training sample. For example, the electronic device may obtain multiple training samples and then perform transformation operations on each of the multiple training samples according to steps e1-e2 above to obtain training samples after the transformation operations (images or true values ​​at different perspectives) for each training sample. Ultimately, the first training sample and the second training sample are combined to form a training sample set.

[0128] Step S302: The electronic device trains a preset model to be trained based on the training sample set to obtain an image detection model.

[0129] Among them, the structure of the model to be trained can refer to the structure of the above-mentioned image detection model, and will not be repeated here.

[0130] In one possible implementation, as described above, the training sample set may include multiple training samples. The electronic device may input one or more training samples into the model to be trained each time to obtain a predicted value (prediction mask) output by the model to be trained, and then calculate a loss function based on the predicted value and the true value of the training query, and adjust the parameters of the model to be trained according to the loss function until the model to be trained converges.

[0131] Optionally, the conditions for the convergence of the model to be trained (or the completion of training) may include: the number of times the electronic device inputs the training sample into the model to be trained reaches a preset number threshold, or the error between the predicted value predicted by the model to be trained and the true value of the training query is less than a preset error threshold, etc. The embodiments of the present application are not limited to this.

[0132] The number threshold can be preset in the electronic device by the administrator. For example, the number threshold can be set to 1,000, 5,000, or 10,000 times. The present embodiment of the application does not limit the specific value of the number threshold. The error threshold can also be preset in the electronic device by the administrator. For example, the error threshold can be set to 2%, 5%, or 10%. The present embodiment of the application does not limit the specific value of the error threshold.

[0133] In another possible implementation, as described above, the model to be trained (image detection model) may include a feature encoding network, an information aggregation network, and a decoding network. Since the feature encoding network can use a sufficiently powerful visual base model, the information aggregation network and the decoding network can be trained first and then trained together. In this case, step S302 may include the following steps f1 to f3.

[0134] Step f1: Based on the training sample set, the information aggregation network and the decoding network are trained until the information aggregation network and the decoding network converge.

[0135] Optionally, as described above, the training sample set may include multiple training samples. The electronic device may input one or more training samples into a fixed feature encoding network each time to obtain a feature map output by the feature encoding network, and further input the feature map into the information aggregation network for aggregation to obtain an aggregated feature map, and then input the aggregated feature map into the decoding network to obtain a prediction result output by the decoding network, and then calculate a loss function based on the prediction result and the true value of the training query, and adjust the parameters in the information aggregation network and the decoding network according to the loss function until the information aggregation network and the decoding network converge.

[0136] For example, taking the industrial product as a wooden board and the detection task as detecting scratches on the wooden board, the training sample includes a first wooden board image (training query image), a first mask representing the position and shape of the scratch in the first wooden board image (training query truth value), a second wooden board image (training prompt image), and a second mask representing the position and shape of the scratch in the second wooden board image (training prompt truth value). The electronic device can input the training sample into the feature encoding network, extract the query feature map representing the query feature of the first wooden board image, and the prompt feature map representing the prompt feature of the second wooden board image, respectively, and input the query feature map, the prompt feature map, and the second mask into the information aggregation network for information aggregation to obtain an aggregated feature map, and then decode the aggregated feature map through the decoding network and upsample it until it is mapped to the size of the first wooden board image to obtain a prediction result (a prediction mask representing the predicted scratch position and shape), and then calculate the loss function based on the prediction result and the first mask, and adjust the parameters in the information aggregation network and the decoding network according to the loss function.

[0137] For another example, taking the industrial product as a circuit board and the detection task as predicting the key welding points of the circuit board, the training sample includes a first circuit board image (training query image), a third mask (training query truth value) representing the position and shape of the key welding points in the first circuit board, a second circuit board image (training prompt image), and a fourth mask (training prompt truth value) representing the position and shape of the key welding points in the second circuit board. The electronic device can input the training sample into the feature encoding network, extract the query feature map representing the query feature of the first circuit board image, and the prompt feature map representing the prompt feature of the second circuit board, and input the query feature map, the prompt feature map, and the fourth mask into the information aggregation network for information aggregation to obtain an aggregated feature map, and then decode the aggregated feature map through the decoding network and upsample it until it is mapped to the size of the first circuit board image to obtain a prediction result (a prediction mask representing the predicted key welding point position and shape), and then calculate the loss function based on the prediction result and the third mask, and adjust the parameters in the information aggregation network and the decoding network according to the loss function.

[0138] Optionally, the conditions for the information aggregation network and the decoding network to converge (or terminate training) may include: the number of times the electronic device inputs the feature map into the information aggregation network reaches a first number threshold and / or the number of times the aggregated feature map is input into the decoding network reaches a second number threshold, or the error between the prediction result predicted by the decoding network and the true value of the training query is less than an error threshold, etc. This application is not limited to this.

[0139] Step f2: Obtain a candidate model based on the converged information aggregation network, decoding network, and feature encoding network.

[0140] Step f3: Train the candidate model based on the training sample set to obtain an image detection model.

[0141] In another possible implementation, as described above, before using the image detection model to predict the query truth value, the electronic device can also map the prompt truth value to a d-dimensional task space. To avoid overfitting the mapping relationship between elements and channels of different categories, the truth values ​​(prompt truth value and query truth value) can be randomly mapped to different channels during training. In this case, the training prompt truth value and the training query truth value are both training masks, and the training mask includes different elements. The above step S301 can specifically include the following steps g1 to g4.

[0142] Step g1: The electronic device randomly maps each type of element in the training mask to a channel of any dimension in the d-dimensional task space.

[0143] Step g2: The electronic device inputs the training sample into the model to be trained to obtain the initial prediction result output by the training model.

[0144] In step g3, the electronic device reversely maps the initial prediction result from the d-dimensional task space to the mode of the training mask to obtain the target prediction result, thus completing one training.

[0145] Step g4: The electronic device trains the model to be trained multiple times to obtain an image detection model.

[0146] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art will appreciate that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical objectives can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0147] In an exemplary embodiment, an image detection device is provided. FIG11 is a schematic diagram showing the composition of the image detection device. As shown in FIG11 , the image detection device includes an acquisition module 1101 and a processing module 1102 .

[0148] Acquisition module 1101 is used to obtain a query image and one or more prompt information pairs, each prompt information pair includes a prompt image and a prompt true value corresponding to the prompt image; the one or more prompt information pairs are used to indicate the type of the first detection task; the prompt true value is used to represent the result obtained after performing the first detection task on the corresponding prompt image.

[0149] Processing module 1102 is used to input the query image and one or more prompt information pairs into the image detection model, and perform a first detection task on the query image based on the prompt image and the corresponding prompt true value through the image detection model to obtain a detection result; the image detection model has the function of performing different types of detection tasks according to different prompt information pairs; different types of detection tasks include the first detection task.

[0150] In some embodiments, the image detection model includes a feature encoding network, an information aggregation network, and a decoding network; the processing module 1102 is specifically used to extract the query feature map of the query image and the prompt feature map of one or more prompt images through the feature encoding network; the size of the query feature map and the prompt feature map is smaller than the size of the query image; the query feature map, the prompt feature map of one or more prompt images, and the prompt true value corresponding to one or more prompt images are aggregated through the information aggregation network to obtain an aggregated feature map; the aggregated feature map is decoded and upsampled through the decoding network until it is mapped to the size of the query image to obtain the detection result of the first detection task.

[0151] In other embodiments, the prompt true value is a mask; the processing module 1102 is also used to map the mask to a d-dimensional task space after the acquisition module 1101 acquires the query image and one or more prompt information pairs; d is a positive integer greater than 1; after performing the first detection task on the query image through a preset image detection model and obtaining the detection result, the detection result is reversely mapped from the d-dimensional task space to the modality of the mask.

[0152] In some further embodiments, each dimension of the d-dimensional task space corresponds to a channel; the information aggregation network includes multiple cross-attention units, and each cross-attention unit includes an attention layer and a feedforward network; the processing module 1102 is specifically used to splice the prompt feature map and the prompt true value corresponding to each prompt image in the channel dimension, and map them into the first feature map and the second feature map corresponding to each prompt image through a convolutional network; each cross-attention unit performs a first operation on the query feature and the first feature map and the second feature map corresponding to a prompt image to obtain a feature map corresponding to a prompt image output by each cross-attention unit; the first operation includes: aggregating the query feature and the first feature map and the second feature map corresponding to a prompt image through the attention layer, and encoding the aggregated feature map through the feedforward network; feature averaging is performed on the feature maps corresponding to one or more prompt images output by multiple cross-attention units to obtain an aggregated feature map.

[0153] In some other embodiments, the different types of detection tasks performed by the image detection model include edge extraction tasks, semantic segmentation tasks, and key point prediction tasks; for the edge extraction task, the mask includes a foreground mask; the processing module 1102 is specifically used to set the elements of each dimension of the d-dimensional task space to 0; replace the foreground mask with a channel of any dimension in the d-dimensional task space; for the semantic segmentation task, the mask includes targets of different categories; the processing module 1102 is specifically used to replace targets of different categories in the mask with channels of different dimensions in the d-dimensional task space; for the key point prediction task, the mask includes key points of different categories; the processing module 1102 is specifically used to replace key points of different categories in the mask with channels of different dimensions in the d-dimensional task space.

[0154] In some other embodiments, the acquisition module 1101 is also used to obtain a training sample set; the training sample set includes training samples corresponding to multiple detection tasks; each training sample includes a training query image, a training query true value corresponding to the training query image, and a training prompt information pair, the training prompt information pair includes a training prompt image, and a training prompt true value corresponding to the training prompt image, the training prompt true value is used to indicate the detection result obtained after performing the training detection task on the corresponding training prompt image; the training detection task is the detection task corresponding to the training sample where the training prompt true value is located; the processing module 1102 is also used to train the preset model to be trained based on the training sample set to obtain an image detection model.

[0155] In some other embodiments, the training query image and the training prompt image in each training sample are the same, and the training query true value and the training prompt true value are the same.

[0156] In some other embodiments, the acquisition module 1101 is specifically used to obtain the training image, and the original prompt true value and the original query true value of the training image; the original prompt true value and the original query true value are both original masks; each time the following operation is performed: the local content of the original mask is used as the training prompt true value and the training query true value corresponding to the training image, and multiple training prompt true values ​​and multiple training query true values ​​corresponding to the training image are obtained; each time the following operation is performed: according to the training image, the training prompt true value corresponding to the training image, and the training query true value corresponding to the training image, the training sample corresponding to the training image is obtained, and multiple training samples corresponding to the training image are obtained; according to the multiple training samples corresponding to the training image, a training sample set is obtained.

[0157] In some further embodiments, the acquisition module 1101 is specifically used to acquire a first training sample; the first training sample includes a first information pair and a second information pair; the first information pair includes a first training query image and a training query true value corresponding to the first training query image; the second information pair includes a first training prompt image and a training prompt true value corresponding to the first training prompt image; a transformation operation is performed on the first information pair and / or the second information pair to obtain a second training sample; the transformation operation includes any one or more of the following: rotation, scaling, translation, cropping, flipping, adding noise, and color transformation; a training sample set is obtained based on the first training sample and the second training sample.

[0158] In some other embodiments, the training prompt true value and the training query true value are training masks, and the training mask includes elements of different types; the processing module 1102 is specifically used to randomly map the elements of each type in the training mask to a channel of any dimension in the d-dimensional task space; the training sample is input into the model to be trained to obtain the initial prediction result output by the model to be trained; the initial prediction result is reversely mapped from the d-dimensional task space to the mode of the training mask to obtain the target prediction result, and a training is completed; the model to be trained is trained multiple times to obtain an image detection model.

[0159] In some other embodiments, the model to be trained includes a feature encoding network, an information aggregation network, and a decoding network; the processing module 1102 is specifically used to train the information aggregation network and the decoding network based on the training sample set until the information aggregation network and the decoding network converge; a candidate model is obtained based on the converged information aggregation network and the decoding network, and the feature encoding network; the candidate model is trained based on the training sample set to obtain an image detection model.

[0160] It should be noted that the module division shown in FIG11 is schematic and represents only one logical functional division. In actual implementation, other divisions may be employed. For example, two or more functions may be integrated into a single processing module. Such integrated modules may be implemented in either hardware or software functional units.

[0161] In an exemplary embodiment, a readable storage medium is further provided, including software instructions, which, when executed on an electronic device, enables the electronic device to execute any one of the methods provided in the above embodiments.

[0162] In an exemplary embodiment, a computer program product including computer-executable instructions is further provided. When the computer program product is run on an electronic device, the electronic device is enabled to execute any one of the methods provided in the above embodiments.

[0163] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer-executable instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), etc.

[0164] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple components. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0165] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

[0166] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection of the claims.

Claims

1. An image detection method, characterized in that: The method comprises: Obtaining a query image and one or more prompt information pairs, each prompt information pair including a prompt image and a prompt truth value corresponding to the prompt image; the one or more prompt information pairs are used to indicate the type of a first detection task; the prompt truth value is used to represent a result obtained after performing the first detection task on the corresponding prompt image; The query image and the one or more prompt information pairs are input into an image detection model, and the image detection model performs the first detection task on the query image based on the one or more prompt images and the prompt true value corresponding to each prompt image to obtain a detection result; the image detection model has the function of performing different types of detection tasks according to different prompt information pairs; the different types of detection tasks include the first detection task.

2. The method according to claim 1, characterized in that The image detection model includes a feature encoding network, an information aggregation network, and a decoding network; performing the first detection task on the query image based on the one or more prompt images and the prompt truth value corresponding to each prompt image by the image detection model to obtain a detection result, including: extracting a query feature map of the query image and a prompt feature map of each of the one or more prompt images through the feature encoding network; wherein the query feature map and the prompt feature map of each of the one or more prompt images are both smaller than the size of the query image; Aggregating the query feature map, the prompt feature maps of the one or more prompt images, and the prompt true values ​​corresponding to the one or more prompt images through the information aggregation network to obtain an aggregated feature map; The aggregated feature map is decoded and upsampled by the decoding network until it is mapped to the size of the query image, thereby obtaining a detection result of the first detection task.

3. The method according to claim 2, characterized in that The hint truth value is a mask; After obtaining the query image and one or more prompt information pairs, the method further includes: Mapping the mask to a d-dimensional task space; d is a positive integer greater than 1; After performing the first detection task on the query image based on the one or more prompt images and the prompt truth value corresponding to each prompt image by the image detection model to obtain a detection result, the method further includes: The detection results are reversely mapped from the d-dimensional task space to the modality of the mask.

4. The method according to claim 3, characterized in that Each dimension of the d-dimensional task space corresponds to a channel; the information aggregation network includes a plurality of cross-attention units, each cross-attention unit includes an attention layer and a feedforward network; the query feature map, the prompt feature maps of the one or more prompt images, and the prompt true values ​​corresponding to the one or more prompt images are aggregated by the information aggregation network to obtain an aggregated feature map, including: For each of the one or more prompt images, concatenate the prompt feature map and the prompt true value corresponding to the prompt image in the channel dimension, and map the concatenation result into the first feature map and the second feature map corresponding to the prompt image through a convolutional network; For each of the multiple cross-attention units, a first operation is performed on the query feature and a first feature map and a second feature map corresponding to one of the one or more prompt images by the cross-attention unit to obtain a feature map corresponding to the prompt image output by the cross-attention unit; the first operation includes: aggregating the query feature and the first feature map and the second feature map corresponding to the prompt image through an attention layer of the cross-attention unit, and encoding the aggregated feature map through a feedforward network of the cross-attention unit; Feature averaging is performed on the feature maps corresponding to the one or more prompt images output by the multiple cross-attention units to obtain the aggregated feature map.

5. The method according to claim 3, characterized in that The different types of detection tasks include edge extraction tasks, semantic segmentation tasks, and key point prediction tasks; For the edge extraction task, the mask includes a foreground mask; and mapping the mask to a d-dimensional task space includes: Set the elements of each dimension of the d-dimensional task space to 0; Replace the foreground mask to a channel of any dimension in the d-dimensional task space; For the semantic segmentation task, the mask includes targets of different categories; and mapping the mask to a d-dimensional task space includes: Replacing targets of different categories in the mask to channels of different dimensions in the d-dimensional task space; For the key point prediction task, the mask includes key points of different categories; and mapping the mask to a d-dimensional task space includes: The key points of different categories in the mask are replaced with channels of different dimensions in the d-dimensional task space.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Acquire a training sample set; the training sample set includes training samples corresponding to multiple detection tasks; each training sample includes a training query image, a training query truth value corresponding to the training query image, and a training prompt information pair, the training prompt information pair includes a training prompt image and a training prompt truth value corresponding to the training prompt image, the training prompt truth value is used to indicate a detection result obtained after performing a training detection task on the corresponding training prompt image; the training detection task is the detection task corresponding to the training sample where the training prompt truth value is located; Based on the training sample set, the preset model to be trained is trained to obtain the image detection model.

7. The method according to claim 6, characterized in that The training query image and the training prompt image in each training sample are the same, and the training query truth value and the training prompt truth value are the same.

8. The method according to claim 7, characterized in that The obtaining of the training sample set includes: Obtaining a training image, and an original hint true value and an original query true value of the training image; the original hint true value and the original query true value are both original masks; Each time the following operation is performed: using the local content of the original mask as the training hint true value and the training query true value corresponding to the training image, to obtain multiple training hint true values ​​and multiple training query true values ​​corresponding to the training image; Each time the following operation is performed: obtaining a training sample corresponding to the training image according to the training image, a training prompt true value corresponding to the training image, and a training query true value corresponding to the training image, thereby obtaining a plurality of training samples corresponding to the training image; The training sample set is obtained according to a plurality of training samples corresponding to the training image.

9. The method according to claim 6, characterized in that The obtaining of the training sample set includes: Acquire a first training sample; the first training sample includes a first information pair and a second information pair; the first information pair includes a first training query image and a training query truth value corresponding to the first training query image; the second information pair includes a first training prompt image and a training prompt truth value corresponding to the first training prompt image; Performing a transformation operation on the first information pair and / or the second information pair to obtain a second training sample; the transformation operation includes any one or more of the following: rotation, scaling, translation, cropping, flipping, adding noise, and color transformation; The training sample set is obtained according to the first training sample and the second training sample.

10. The method according to claim 6, characterized in that The training prompt true value and the training query true value are training masks, and the training mask includes elements of different types; The step of training a preset model to be trained based on the training sample set to obtain the image detection model includes: Randomly map each type of element in the training mask to a channel of any dimension in the d-dimensional task space; Inputting the training sample into the model to be trained to obtain an initial prediction result output by the model to be trained; Reversely mapping the initial prediction result from the d-dimensional task space to the modality of the training mask to obtain a target prediction result, completing one training; The model to be trained is trained multiple times to obtain the image detection model.

11. The method according to claim 6, characterized in that The model to be trained includes a feature encoding network, an information aggregation network, and a decoding network; the preset model to be trained is trained based on the training sample set to obtain the image detection model, including: Training the information aggregation network and the decoding network based on the training sample set until the information aggregation network and the decoding network converge; Obtaining a candidate model based on the converged information aggregation network and decoding network, and the feature encoding network; The candidate model is trained based on the training sample set to obtain the image detection model.

12. A readable storage medium, characterized in that: The storage medium includes: software instructions; when the software instructions are executed in an electronic device, the electronic device implements the method according to any one of claims 1 to 11.

13. An image detection device, comprising: An acquisition module, configured to acquire a query image and one or more prompt information pairs, each prompt information pair including a prompt image and a prompt truth value corresponding to the prompt image; The one or more prompt information pairs are used to indicate the type of the first detection task; the prompt truth value is used to indicate the result obtained after performing the first detection task on the corresponding prompt image; A processing module is used to input the query image and the one or more prompt information pairs into an image detection model, and perform the first detection task on the query image based on the one or more prompt images and the prompt true value corresponding to each prompt image through the image detection model to obtain a detection result; the image detection model has the function of performing different types of detection tasks according to different prompt information pairs; the different types of detection tasks include the first detection task.

14. An electronic device comprising a processor and a memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the method according to any one of claims 1 to 11 is implemented. 15 . A computer program product, which, when executed in an electronic device, enables the electronic device to execute the method according to claim 1 .

Images