Image detection method, device, equipment, and computer program

Abstract

The present invention discloses an image detection method, device, equipment, and readable storage medium, which includes the steps of: performing a feature extraction process on an image to be detected to obtain at least two sub-image features; generating attention weights corresponding to each of the at least two sub-image features; performing a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; performing a clustering sampling process on the at least two sub-image features to obtain sampled sub-image features contained in each of the at least two classification clusters; determining block sparse self-attention corresponding to each sampled sub-image feature based on the at least two classification clusters and a block sparse matrix; determining a second feature vector based on the at least two block sparse self-attention; and determining a classification result of the image to be detected based on the first feature vector and the second feature vector. The present invention can improve the detection speed and detection accuracy of images.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This application relates to the field of computer technology, and more particularly to image detection technology.

[0002] This application claims priority to a Chinese patent application filed with the China Patent Office on March 23, 2022, bearing application number 2022102886990 and entitled "IMAGE DETECTION METHOD, APPARATUS, DEVICE, AND READABLE STORAGE MEDIUM", the entire contents of which are incorporated herein by reference. [Background technology]

[0003] In traditional machine learning image classification problems, it is often assumed that an image clearly belongs to a certain category. However, in real applications, multiple instances are observed in an image, but the image label only describes the category of one of the instances. Such problems are generally called multiple instance learning (MIL). The purpose of multiple instance learning is to create a multi-instance learning model by learning multi-instance images with classification labels, and then apply the multi-instance learning model to detect unknown multi-instance images.

[0004] In the related art, a self-attention module mines information of all instances in an existing multi-instance image, finds related information between instances therein, and creates a multi-instance learning model, so that unknown multi-instance images can be detected. However, the self-attention module has high computational complexity, and when modeling a multi-instance image such as a digitized pathology image in which the number of instances can reach about 10,000, hardware resources and time are consumed, making training difficult. And for multi-instance images such as digitized pathology images, which are small in number and each image corresponds only to the label of one whole image, the supervision information is very weak, and when training a self-attention module with such high complexity on a small data set, it is difficult to ensure that the self-attention module can mine effective information, and the self-attention module is prone to overfitting problems, causing low detection accuracy. Summary of the Invention [Means for solving the problem]

[0005] The embodiments of the present application provide an image detection method, device, apparatus, and readable storage medium, which can improve the image detection speed and detection accuracy.

[0006] In one aspect, embodiments of the present disclosure provide a method for image detection performed by a computing device, the method comprising: acquiring an image of the target object, and performing a feature extraction process on the image of the target object to obtain a feature representation subset of the image of the target object, the image of the target object including at least two sub-images, the feature representation subset including at least two sub-image features, the at least two sub-image features having a one-to-one correspondence with the at least two sub-images; generating attention weights corresponding to the at least two sub-image features, and performing a weighted aggregation process on the at least two sub-image features according to the attention weights to obtain a first feature vector; performing a clustering sampling process on at least two sub-image features to obtain at least two classification clusters, the classification clusters including the sampled sub-image features; determining block sparse self-attention corresponding to each sampled sub-image feature based on the at least two classification clusters and the block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions, where the block sparse self-attention corresponding to the sampled sub-image feature is determined based on the sampled sub-image feature in the classification cluster to which it belongs; and determining a classification result for the image to be detected based on the first feature vector and the second feature vector.

[0007] In one aspect, embodiments of the present disclosure provide a method for image detection performed by a computing device, the method comprising: obtaining a sample image and performing a feature extraction process on the sample image to obtain a sample feature representation subset of the sample image, the sample image including at least two sample sub-images, the sample feature representation subset including at least two sample sub-image features, the at least two sample sub-image features having a one-to-one correspondence with the at least two sample sub-images; inputting at least two sample sub-images into an initial image discrimination model, generating sample attention weights corresponding to each of the at least two sample sub-image features through the initial image discrimination model, and performing a weighted aggregation process on the at least two sample sub-image features according to the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector; performing a clustering sampling process on at least two sample sub-image features using an initial image identification model to obtain at least two sample classification clusters, the sample classification clusters including the sample sampling sub-image features; determining sample block sparse self-attentions corresponding to each sample sampling sub-image feature based on the at least two sample classification clusters and the block sparse matrix; and determining a second sample feature vector based on the at least two sample block sparse self-attentions, where the sample block sparse self-attentions corresponding to the sample sampling sub-image features are determined based on the sample sampling sub-image features in the sample classification cluster to which the sample sampling sub-image features belong; determining a sample classification result of the sample image based on the first sample feature vector and the second sample feature vector according to an initial image classification model; The method includes a step of adjusting model parameters for the initial image identification model based on the at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and the classification labels corresponding to the sample images, to obtain an image identification model used to identify the classification results of the image to be detected.

[0008] In one aspect, the present embodiments provide an image detection apparatus, comprising: a feature extraction module for acquiring an image of a target object, performing a feature extraction process on the image of the target object, and obtaining a feature representation subset of the image of the target object, the image of the target object including at least two sub-images, the feature representation subset including at least two sub-image features, the at least two sub-image features having a one-to-one correspondence with the at least two sub-images; a first vector generating module, which is used for generating attention weights corresponding to at least two sub-image features, and performing a weighted aggregation process on the at least two sub-image features according to the attention weights to obtain a first feature vector; A second vector generation module is used for performing a clustering sampling process on at least two sub-image features to obtain at least two classification clusters, the classification clusters including the sampled sub-image features, determining block sparse self-attention corresponding to each sampled sub-image feature based on the at least two classification clusters and a block sparse matrix, and determining a second feature vector based on the at least two block sparse self-attentions, where the block sparse self-attention corresponding to the sampled sub-image feature is determined based on the sampled sub-image feature in the classification cluster to which it belongs; A classification module is used for determining a classification result of the image to be detected based on the first feature vector and the second feature vector.

[0009] In one aspect, the present embodiments provide an image detection apparatus, comprising: a sample feature extraction module for obtaining a sample image, performing a feature extraction process on the sample image, and obtaining a sample feature representation subset of the sample image, the sample image including at least two sample sub-images, the sample feature representation subset including at least two sample sub-image features, the at least two sample sub-image features having a one-to-one correspondence with the at least two sample sub-images; a first sample vector generation module, which is used to input at least two sample sub-images into an initial image discrimination model, generate sample attention weights corresponding to each of the at least two sample sub-image features through the initial image discrimination model, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector; a second sample vector generation module for performing a clustering sampling process on at least two sample sub-image features according to the initial image discrimination model to obtain at least two sample classification clusters, the sample classification clusters including the sample sampling sub-image features; determining sample block sparse self-attentions corresponding to each sample sampling sub-image feature according to the at least two sample classification clusters and the block sparse matrix; and determining a second sample feature vector according to the at least two sample block sparse self-attentions, where the sample block sparse self-attentions corresponding to the sample sampling sub-image features are determined according to the sample sampling sub-image features in the sample classification cluster to which the sample sampling sub-image features belong; A sample classification module is used for determining a sample classification result of the sample image according to the first sample feature vector and the second sample feature vector according to the initial image identification model; and a training module used for adjusting model parameters for the initial image identification model based on the at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and the classification labels corresponding to the sample images, to obtain an image identification model used to identify the classification results of the image to be detected.

[0010] In one aspect, the present embodiment provides a computing device including a processor, a memory, and a network interface, The processor is connected to the memory and the network interface, where the network interface is used to provide a data communication network element, the memory is used to store a computer program, and the processor is used to call the computer program and thereby perform the method in the present embodiment.

[0011] In one aspect, the present embodiment provides a computer-readable storage medium having a computer program stored therein, the computer program being suitable for being loaded by a processor and for executing a method in the present embodiment.

[0012] In one aspect, the present embodiment provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium, the computer instructions being read by a processor of a computing device from the computer-readable storage medium, and the processor executing the computer instructions to cause the computing device to perform a method in the present embodiment.

[0013] In the embodiment of the present application, a feature extraction process is performed on a target image including at least two sub-images, and a feature representation subset of the target image can be obtained, the feature representation subset including sub-image features corresponding to each of the at least two sub-images; then, information of the sub-images is mined in two ways: the first is to independently mine information of each sub-image, that is, to generate attention weights corresponding to each of the at least two sub-image features, and perform a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; the second is to mine related information between sub-images of the same category, that is, to perform a clustering sampling process on the at least two sub-image features to obtain sampled sub-image features included in each of the at least two classification clusters, and determine block sparse self-attention corresponding to each sampled sub-image feature based on the at least two classification clusters and a block sparse matrix, and determine a second feature vector based on the at least two block sparse self-attention; and finally, to determine a classification result of the target image based on the first feature vector and the second feature vector. By adopting the method provided in the embodiments of the present application, the first feature vector and the second feature vector obtained by the two information mining methods can complement and constrain each other, thereby improving the accuracy of image detection; and by calculating the block sparse self-attention corresponding to the sampled sub-image features by a block sparse matrix, it is possible to ensure that only the relationship between the sampled sub-image features and the sampled sub-image features belonging to a common classification cluster is paid attention to, thereby reducing the computational complexity and improving the detection speed. [Brief description of the drawings]

[0014] [Figure 1a] FIG. 1 is a schematic diagram of a network architecture provided by an embodiment of the present application. [Figure 1b] FIG. 2 is a schematic diagram illustrating an application scene of the image detection method provided in the present embodiment. [Diagram 2]FIG. 2 is a flow diagram of an image detection method provided in the present embodiment. [Diagram 3] FIG. 2 is a scene schematic diagram of an image feature extraction process provided by an embodiment of the present application. [Figure 4] FIG. 2 is a flow diagram of an image detection method provided in the present embodiment. [Diagram 5] FIG. 1 is a schematic diagram of the clustering results of colorectal pathology images provided in an example of the present application. [Figure 6] FIG. 1 is a schematic diagram of the principle of applying block sparse constraints to global self-attention provided by an embodiment of the present application; [Figure 7] FIG. 2 is a structural schematic diagram of an image recognition model provided in the present embodiment. [Figure 8] FIG. 2 is a flow diagram of a method for training an initial image recognition model provided by an embodiment of the present application. [Figure 9] FIG. 2 is a structural schematic diagram of an image detection device provided in the present embodiment; [Figure 10] FIG. 2 is a structural schematic diagram of a computer device provided by an embodiment of the present application; [Figure 11] FIG. 2 is a structural schematic diagram of another image detection device provided by an embodiment of the present application; [Figure 12] FIG. 2 is a structural schematic diagram of another computer device provided by an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The following clearly and completely describes the technical solutions in the embodiments of the present application in combination with the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without requiring creative work are all within the scope of protection of the present application.

[0016] The solutions provided by the embodiments of the present application relate to artificial intelligence computer vision technology, machine learning, deep learning and other technologies, which are specifically described in the following embodiments.

[0017] Referring to Fig. 1a, Fig. 1a is a schematic diagram of a network architecture provided by an embodiment of the present application. As shown in Fig. 1a, the network architecture may include a service server 100 and a group of terminal devices, the group of terminal devices may include terminal devices 10a, 10b, 10c, ... and 10n, where any terminal device in the group of terminal devices can have a communication connection with the service server 100, for example, a communication connection exists between the terminal device 10a and the service server 100, a communication connection exists between the terminal device 10b and the service server 100, and a communication connection exists between the terminal device 10c and the service server 100, where the above communication connection is not limited to a connection method, and may be directly or indirectly connected by a wired communication method, directly or indirectly connected by a wireless communication method, or may be connected by other methods, and the present application does not limit it here.

[0018] It should be understood that each terminal device in the terminal group shown in Fig. 1a may have an application client terminal installed therein, and when the application client terminal is operated in each terminal device, it can exchange data with the service server 100 shown in Fig. 1a, so that the service server 100 can receive service data from each terminal device. Here, the application client terminal may be an application client terminal having a game application, a video editing application, a social application, an instant communication application, a live broadcast application, a short video application, a video application, a music application, a shopping application, a novel application, a payment application, or a related image processing function such as a browser. Here, the application client terminal may be an independent client terminal, or an embedded sub-client terminal integrated in a client terminal (such as an instant communication client terminal, a social client terminal, a video client terminal, etc.), and is not limited here.

[0019] As shown in Fig. 1a, each terminal device in the terminal device group can obtain a detection target image and transmit it as service data to the service server 100 by operating the application client terminal, and the service server 100 can perform image detection on the detection target image, thereby determining a classification result of the detection target image. Here, the detection target image can also be referred to as a multi-instance image, i.e., includes at least two sub-images, and one sub-image can be referred to as one instance, and for a multi-instance image, if one instance is abnormal, the multi-instance image can be regarded as an abnormal image, in other words, if one sub-image is abnormal, the classification result of the detection target image should be an abnormal image.

[0020] In one possible embodiment, the image to be detected is taken as a digitized pathology image, which can be obtained in the following manner: A fully automatic microscope or an optical magnification system is used to scan a carrier slice to collect a high-resolution digital image, and then a computer is used to automatically perform high-precision multi-view seamless splicing and processing on the obtained high-resolution digital image, thereby obtaining high-quality visualization data, that is, a digitized pathology image. The digitized pathology image can be enlarged and reduced at any position in a computer device, and does not have the problems of image information distortion and details becoming unclear, which makes it more convenient for doctors to perform pathological diagnosis such as cancer diagnosis, survival stage detection, and gene mutation detection, compared with the original carrier slice observation. However, because digitized pathology images have high resolution, are often very large in image size, and contain a large number of instances (biological tissues such as cells and genes), it is often necessary to continuously adjust the detection position and detection magnification when artificially observing the digitized pathology images, which requires a large amount of time and effort. Therefore, the digitized pathology image is loaded by the above-mentioned application client terminal, and after the terminal device obtains the digitized pathology image, it can be transmitted to the service server 100 as service data. Furthermore, the service server 100 can perform image detection on the digitized pathology image and determine a classification result of the digitized pathology image, and the classification result can assist doctors in making medical diagnoses.

[0021] After obtaining the image of the detection target, the service server 100 can perform image detection on the image of the detection target and determine the classification result of the image of the detection target. The specific implementation process is also referred to in FIG. 1b, which is a schematic diagram of the application scene of the image detection method provided in the embodiment of the present application. For ease of understanding, the detection target image is still taken as an example of the digitized pathology image in the above embodiment. As shown in FIG. 1b, a patient management application 300 is installed in a terminal device 200 (which may be any terminal device in the above FIG. 1a, for example, the terminal device 10a), and a subject A has an associated relationship with the terminal device 200. Assuming that the subject A is the doctor in charge of the subject B, the subject A can check the case material of the subject B, for example, the colorectal pathology image 301, on the patient management application 300, and by observing the colorectal pathology image 301, the subject A can diagnose whether the subject B is suffering from colorectal cancer. Because the image size of the colorectal pathology image 301 is very large and there are many cellular tissues that need to be observed, it takes a long time for the subject A to manually observe it. Therefore, the subject A can initiate an image detection request for the colorectal pathology image 301 to the service server 400 (e.g., the service server 100 shown in FIG. 1a above) through the patient management application 300 running on the terminal device 200, and then the service server 400 can perform image detection on the colorectal pathology image 301 and determine the classification result of the colorectal pathology image 301, that is, determine whether the colorectal pathology image 301 belongs to a normal image or an abnormal image. The classification result of the colorectal pathology image 301 by the service server 400 can assist the subject A to make a disease diagnosis of the subject B.

[0022] As can be understood, the colorectal pathology image 301 has a large image size and contains a large amount of cellular tissue, and therefore can be considered to contain at least two sub-images in the colorectal pathology image 301 (i.e., at least two sub-images can be obtained by dividing the colorectal pathology image 301), and if an abnormality is present in one sub-image, the colorectal pathology image 301 is an abnormal image.

[0023] As shown in FIG. 1b, after the terminal device 200 sends a colorectal pathology image 301 to the service server 400, the service server 400 can first perform a feature extraction process on the colorectal pathology image 301 to obtain a feature representation subset 401 used to represent the colorectal pathology image 301, where the feature representation subset 401 includes at least two sub-image features, and each sub-image feature is used to describe the information of one sub-image in the colorectal pathology image 301. Then, the service server 400 can detect the colorectal pathology image 301 using an image identification model 402, which may include a first attention sub-network 4021, a second attention sub-network 4022, and a classification sub-network 4023, where the first attention sub-network 4021 is used for regarding the sub-images as independent instances, and mining independent representation information of each sub-image based on each sub-image feature, and obtaining a first feature vector used for representing the independent representation information; the second attention sub-network 4022 is used for mining global representation information among all sub-images, and obtaining a second feature vector used for representing the global representation information; and the classification sub-network 4023 is used for performing image classification on the image to be detected based on the first feature vector and the second feature vector, and determining a classification result of the image to be detected.

[0024] As shown in Fig. 1b, after the service server 400 inputs the feature representation subset 401 into the image identification model 402, the first attention sub-network 4021 generates attention weights corresponding to at least two sub-image features, and then performs a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector 403. At the same time, the second attention sub-network 4022 performs a clustering sampling process on the at least two sub-image features to obtain sampled sub-image features contained in at least two classification clusters, and then determines block sparse self-attentions corresponding to each sampled sub-image feature based on the at least two classification clusters and the block sparse matrix, and determines a second feature vector 404 based on the at least two block sparse self-attentions. After determining the first feature vector 403 and the second feature vector 404, the classification sub-network 4023 can perform feature fusion processing on the first feature vector 403 and the second feature vector 404 to obtain a fusion feature vector, and then perform classification processing on the fusion feature vector to obtain a classification result 405 of the colorectal pathology image 301, where the classification result 405 can include the normal probability and abnormal probability of the colorectal pathology image 301, where the normal probability refers to the probability that the colorectal pathology image 301 is a normal image, that is, the probability that the subject B is not affected by a disease, and the abnormal probability refers to the probability that the colorectal image 301 is an abnormal image, that is, the probability that the subject B may be affected by colorectal cancer. The service server 400 will return the classification result 405 to the terminal device 200, and the subject A can diagnose the disease status of the subject B based on the classification result 405.

[0025] Optionally, if the image identification model 402 is stored locally in the terminal device 200, the terminal device 200 can locally perform an image detection task on the image to be detected and obtain a classification result of the image to be detected. Since the training image identification model 402 involves a large amount of offline calculation, the local image identification model of the terminal device 200 may be trained by the service server 400 and then transmitted to the terminal device.

[0026] As can be understood, the method provided by the embodiment of the present application may be executed by a computer device, and the computer device includes, but is not limited to, a terminal device or a server, and the service server 100 in the embodiment of the present application may be a computer device, and the terminal device in the terminal device group may also be a computer device, and is not limited thereto. The server may be an independent physical server, a server group composed of multiple physical servers, or a distributed system, and may further be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, big data, and artificial intelligence platforms. The terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, etc. The embodiment of the present application may be applied to various scenarios, including, but not limited to, cloud technology, cloud security, domain blockchain, artificial intelligence, smart transportation, driving assistance, etc.

[0027] As can be understood, when the embodiments of the present application are applied to a specific product or technology, relevant data, such as images of the relevant detected object, need to be obtained after obtaining user permission or consent, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0028] It is necessary to explain that in the present embodiment, the image to be detected is described as a colorectal pathology image as an example, but in actual application scenarios, the image to be detected may also be a pathology image of other types of cancer, or other multi-instance images including at least two sub-images, and the present application is not limited thereto.

[0029] Further, as shown in FIG. 2, FIG. 2 is a flow diagram of an image detection method provided by an embodiment of the present application. Here, the image detection method may be executed by a computer device, where the computer device may be the service server 100 shown in FIG. 1a above, or any terminal device in the terminal device group shown in FIG. 1a above, for example, the terminal device 10c. Hereinafter, the image detection method will be described by taking an example in which the image detection method is executed by a computer device. Here, the image detection method may include at least the following steps S101 to S104.

[0030] Step S101: Obtain an image of a detection target, perform a feature extraction process on the image of the detection target, and obtain a feature representation subset of the image of the detection target, where the image of the detection target includes at least two sub-images, the feature representation subset includes at least two sub-image features, and the at least two sub-image features have a one-to-one correspondence with the at least two sub-images.

[0031] Specifically, the image to be detected is a weak image annotation, which is a multi-instance image in which multiple instances correspond to one label. A multi-instance image is also called a multi-instance bag, which contains several instances, and an instance may be regarded as a sub-image, but only the bag contains a label and the instance does not contain a label. If the multi-instance bag contains at least one positive instance, the bag is marked as a positive class multi-instance bag (positive bag), and if all the instances in the multi-instance bag are negative instances, the bag is marked as a negative class multi-instance bag (negative bag). The image to be detected may be a digitized pathology image applied in pathology diagnosis, such as cancer diagnosis, survival prediction, gene mutation prediction, etc., and image detection is performed on the digitized pathology image to obtain a classification result of the digitized pathology image, which can assist a doctor to determine a corresponding medical diagnosis result. For example, the digitized pathology image is the colorectal pathology image 301 in FIG. 1b above, and image detection is performed on the colorectal pathology image 301 to obtain a classification result 405, which can assist subject A to determine whether subject B is affected by colorectal cancer.

[0032] Specifically, the image to be detected is media data used by humans and lacks information that computer equipment can understand, so the image to be detected needs to be converted from an unstructured original image into structured information that can be identified and processed by a computer, that is, a scientific abstraction needs to be performed on the image to be detected, a mathematical model of the image to be detected is created, and the mathematical model is used to describe and replace the image to be detected, and the computer equipment needs to perform calculations and operations on the mathematical model to realize the identification of the image to be detected. The mathematical model may be a vector space model, and in this case, the sub-image features corresponding to the sub-images included in the image to be detected may be vectors in the vector space model, and the computer equipment can describe and apply the image to be detected by the feature representation subset constructed by the sub-image features.

[0033] Specifically, if all sub-image features are taken as feature items, it will cause a large amount of calculation, so it is necessary to reduce the number of sub-images to be processed as much as possible without damaging the core information of the image to be detected, thereby simplifying the calculation and improving the speed and efficiency of the image to be detected. Therefore, one possible specific process of performing feature extraction processing on the image to be detected and obtaining a feature representation subset of the image to be detected is as follows: Identify the background region and the foreground region in the image to be detected, then perform image segmentation on the image to be detected based on the background region and the foreground region to obtain the foreground image to be detected, then perform a scaling processing on the foreground image to be detected based on the scaling ratio to obtain the foreground image to be trimmed, and then perform a cropping processing on the foreground image to be trimmed based on the preset length of the sub-image and the preset width of the sub-image to obtain at least two sub-images, and finally perform image feature extraction processing on the at least two sub-images respectively to obtain sub-image features corresponding to each of the at least two sub-images, and determine a feature representation subset of the image to be detected based on the sub-image features corresponding to each of the at least two sub-images. Here, the preset length of the sub-image is less than the length of the foreground image to be cropped, and the preset width of the sub-image is less than the width of the foreground image to be cropped.

[0034] In order to easily understand the specific process of the feature extraction process, please refer to FIG. 3, which is a scene schematic diagram of the image feature extraction process provided by the embodiment of the present application. As shown in FIG. 3, after obtaining the image 3000 of the detection target, the computer device first performs foreground background discrimination on the image 3000 of the detection target, determines the foreground region 3001 and the background region 3002 contained in the image 3000 of the detection target, and then performs image division of the foreground background region to obtain the foreground image 3003 of the detection target. Then, according to the preset length of the sub-image and the preset width of the sub-image, for example, 512*512, the foreground image 3003 of the detection target is cropped to obtain at least two sub-images, namely, sub-image 3004, sub-image 3005, ... and sub-image 3006. Then, the at least two sub-images are input to a feature extractor 3007, and the feature extractor 3007 extracts the sub-image features corresponding to each sub-image, thereby obtaining a feature representation subset 3008. Here, the feature extractor 3007 may be realized by adopting the feature extractor RestNet50 (Residual net 50, residual network), or may be realized using other pre-trained networks, and the present application is not limited here. The foreground image 3003 to be detected can be selectively enlarged or reduced at a set magnification, for example, the foreground image 3003 to be detected is enlarged ten times to obtain a foreground image to be trimmed, and the foreground image to be trimmed is subjected to a trimming process, so that more sub-images can be obtained than when the foreground image 3003 to be detected is subjected to a trimming process, and therefore more sub-image features can be obtained, and the image to be detected can be expressed more finely.

[0035] Step S102: Generate attention weights corresponding to each of the at least two sub-image features, and perform a weighted aggregation process on the at least two sub-image features according to the attention weights to obtain a first feature vector.

[0036] Specifically, the attention weight, which may also be referred to as the attention score, is used to measure the importance of a sub-image feature, and the larger the attention weight, the higher the importance of the corresponding sub-image feature, and the larger the proportion of the corresponding sub-image feature in the final output first feature vector. The attention weight corresponding to each sub-image feature can be obtained by a single network learning using the sub-image feature itself as an input.

[0037] Specifically, after obtaining the attention weights, a weighted aggregation process can be performed on the sub-image features, that is, a weighted sum is obtained for the sub-image features based on the attention weights to obtain a first feature vector.

[0038] Step S103: performing a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, the classification clusters including the sampled sub-image features; determining block sparse self-attention corresponding to each of the sampled sub-image features based on the at least two classification clusters and a block sparse matrix; determining a second feature vector based on the at least two block sparse self-attentions, and the block sparse self-attention corresponding to the sampled sub-image feature is determined based on the sampled sub-image feature in the classification cluster to which it belongs.

[0039] Specifically, in order to better mine the association between each sub-image feature and avoid too high computational complexity, at least two sub-image features can be clustered first, i.e., at least two sub-image features are divided into at least two classification clusters based on the similarity of the sub-image features, and the sub-images corresponding to the sub-image features in one classification cluster belong to the same category of images. Then, some sub-image features from each classification cluster are sampled as sampled sub-image features.

[0040] Specifically, the self-attention of the sampled sub-image features is determined based on a global self-attention weight matrix, where the global self-attention weight matrix is ​​used to characterize the association degree between the sampled sub-image features. Since the sampled sub-image features have been previously classified, when determining the self-attention of a sampled sub-image feature, the computer device only needs to pay attention to the sampled sub-image features that belong to the same classification cluster as the sampled sub-image feature. After determining the global self-attention weight matrix based on the sampled sub-image features, the computer device can obtain a block sparse matrix that matches at least two classification clusters, and filter the global self-attention weight matrix based on the block sparse matrix to obtain a block sparse global self-attention weight matrix, which is used to characterize the association degree between the sampled sub-image features of the same classification cluster. Then, the block sparse self-attention of each sampled sub-image feature can be determined based on the block sparse global self-attention weight matrix, and the computer device can then perform an average pooling process on the block sparse self-attention of all the sampled sub-image features to obtain a second feature vector.

[0041] Step S104: A classification result of the detection target image is determined based on the first feature vector and the second feature vector.

[0042] Specifically, after obtaining the first feature vector and the second feature vector, a MLP (Multilayer Perceptron) classifier can be adopted to predict the first feature vector and the second feature vector, and the classification result can be output.

[0043] The method provided in the present embodiment is adopted to perform feature extraction processing on a target image including at least two sub-images, to obtain a feature representation subset of the target image, the feature representation subset including sub-image features corresponding to each of the at least two sub-images; then, mining information of the sub-images in two ways: first, mining information of each sub-image independently, i.e., generating attention weights corresponding to each of the at least two sub-image features, and performing a weighted aggregation processing on the at least two sub-image features based on the attention weights to obtain a first feature vector; second, mining related information between sub-images of the same category, i.e., performing a clustering sampling processing on the at least two sub-image features to obtain at least two classification clusters, each classification cluster including a sampling sub-image feature; determining block sparse self-attention corresponding to each sampling sub-image feature based on the at least two classification clusters and a block sparse matrix; determining a second feature vector based on the at least two block sparse self-attention; and finally determining a classification result of the target image based on the first feature vector and the second feature vector. The first feature vector and the second feature vector obtained by the two information mining methods can complement and constrain each other, so that the accuracy of image detection can be improved; and when calculating the block sparse self-attention corresponding to the sampled sub-image feature by the block sparse matrix, it can ensure that only the relationship between the sampled sub-image feature and the sampled sub-image feature belonging to a common classification cluster is paid attention to, so as to reduce the computational complexity and improve the detection speed.

[0044] Further, as shown in FIG. 4, FIG. 4 is a flow diagram of an image detection method provided by an embodiment of the present application. Here, the image detection method may be executed by a computer device, where the computer device may be the service server 100 shown in FIG. 1a above, or any terminal device in the terminal device group shown in FIG. 1a above, for example, the terminal device 10c. Hereinafter, the image detection method will be described by taking an example in which the image detection method is executed by a computer device. Here, the image detection method may include at least the following steps S201 to S210.

[0045] Step S201: Obtain an image of a detection target, perform a feature extraction process on the image of the detection target, and obtain a feature representation subset of the image of the detection target, where the image of the detection target includes at least two sub-images, the feature representation subset includes at least two sub-image features, and the at least two sub-image features have a one-to-one correspondence with the at least two sub-images.

[0046] Specifically, for the implementation of step S201, reference may be made to the specific description of step S101 in the embodiment corresponding to FIG. 2 above, and detailed description thereof will be omitted here.

[0047] Specifically, suppose the image to be detected is X, and after the tissue-background segmentation and image cropping, at least two sub-images are obtained, and the image set consisting of at least two sub-images is {x1, x2, ..., x n}, where each sub-image x i is called an instance of an image to be detected. The above process of performing feature extraction processing on at least two sub-images can be expressed by the following equation (1).

number

number

[0048] Step S202: input the at least two sub-image features into a first attention sub-network in an image discrimination model, where the first attention sub-network includes a weight learning network layer and a weight aggregation network layer.

[0049] Step S203: The weight learning network layer performs a weight fitting process on the at least two sub-image features respectively to obtain attention weights corresponding to the at least two sub-image features.

[0050] Specifically, the weight learning network layer can adopt a parameterized neural network to learn attention weights corresponding to sub-image features, where the sub-image features h in the feature representation subset H obtained from the above formula (1) are k The corresponding attention weight α k may be expressed as equation (2).

number

[0051] Step S204: The weighted aggregation network layer performs a weighting process on each sub-image feature based on the attention weight, obtains a weighted sub-image feature corresponding to each of the sub-image features, and performs an aggregation process on the weighted sub-image features corresponding to each of the at least two sub-image features to obtain a first feature vector.

[0052] Specifically, in the weighted aggregation network layer, effective aggregation can be performed on the sub-image features by sampling the nonlinear attention weights of the linear terms, that is, it can be calculated by Equation (3).

number

[0053] Step S205: input the at least two sub-image features into a second attention sub-network in the image discrimination model, where the second attention sub-network includes a clustering sampling network layer, a global self-attention network layer, a self-attention network layer, and an average value pooling network layer.

[0054] Specifically, inputting at least two sub-image features into the first attention sub-network and inputting at least two sub-image features into the second attention sub-network may be performed simultaneously, and the first attention sub-network and the second attention sub-network do not affect each other.

[0055] Step S206: The clustering sampling network layer performs a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, the classification clusters including sampling sub-image features, and the sum of the quantity of the sampling sub-image features included in each of the at least two classification clusters is N, where N is a positive integer less than the quantity of the at least two sub-image features.

[0056] Specifically, in the clustering sampling network layer, the computer device can first perform a clustering process on at least two sub-image features to obtain at least two classification clusters, then obtain a k-th classification cluster in the at least two classification clusters, where k is a positive integer, and the k-th classification cluster includes at least one clustering sub-image feature, and then the computer device can obtain a vector distance between each of the at least one clustering sub-image feature and the cluster center of the k-th classification cluster as a reference distance, and sequentially obtain h clustering sub-image features in the at least one clustering sub-image feature based on the reference distance, and the h clustering sub-image features are the sampling sub-image features included in the k-th classification cluster, where h is a positive integer, and h is less than or equal to the quantity of the at least one clustering sub-image feature.

[0057] As can be understood, the number of classification clusters obtained by performing clustering process on at least two sub-image features is assumed to be p, and each classification cluster has a cluster center, and when sampling each classification cluster, h sub-image features that are closest to the cluster center of the classification cluster can be collected, and finally, a total of N sampled sub-image features can be obtained, where N=p×h. In order to reduce the computational complexity, the default sampling total N=128. After the clustering process and the sampling process, the obtained N sampled sub-image features have diversity and can be approximated as an effective representative of the original at least two sub-image features.

[0058] Specifically, the clustering process may adopt an unsupervised k-means (unsupervised clustering method) clustering method or other clustering methods, and is not limited here. In at least two classification clusters obtained after clustering, the image categories to which the sub-images corresponding to the sub-image features included in each classification cluster belong are the same. For convenience of understanding, please also refer to FIG. 5, which is a schematic diagram of the clustering result of colorectal pathology images provided by the present embodiment. As shown in FIG. 5, the sub-image features corresponding to the sub-images of the common group belong to a common classification cluster, and as can be seen in FIG. 5, all different pathology images can be classified into categories with similar approximate histological structure features. For example, the sub-images in group 1 contain significant cancer tissue, all the sub-images in group 2 are sub-images with abnormal staining or relatively poor imaging quality, the sub-images in group 3 are basically glandular tissue, and all the sub-images in group 4 collect a relatively large number of immune cells, etc.

[0059] Step S207: Determine a block sparse global self-attention weight matrix of the N sampling sub-image features based on the block sparse matrix by the global self-attention network layer.

[0060] Specifically, the second attention sub-network includes a query weight matrix and a key weight matrix. The process of the computer device determining the block sparse global self-attention weight matrix of the N sampled sub-image features based on the block sparse matrix by the global self-attention network layer may be as follows: the global self-attention network layer constructs a sampled sub-image feature matrix according to the sampled sub-image features included in each of at least two classification clusters, multiplies the sampled sub-image feature matrix with the query weight matrix to obtain a query matrix, multiplies the sampled sub-image feature matrix with the key weight matrix to obtain a key matrix, determines a block sparse global relevance matrix according to the query matrix, a transpose matrix corresponding to the key matrix, and the block sparse matrix, and performs a normalization process on the block sparse global relevance matrix to obtain a block sparse global self-attention weight matrix.

[0061] Specifically, the above sampling sub-image feature matrix may be expressed as Equation (4).

number

number

number

number

[0062] Specifically, the calculation of the query matrix can be expressed by equation (5).

number

number

number

[0063] Similarly, the calculation of the key matrix can be expressed by equation (6).

number

number

number

[0064] Therefore, the calculation of the block sparse global self-attention weight matrix can be expressed by Equation (7).

number

[0065] For ease of understanding, please also refer to FIG. 6, which is a schematic diagram of the principle of block sparse constraint on global self-attention provided by the embodiment of the present application. As shown in FIG. 6, the global self-attention weight matrix 601 is a global self-attention matrix when not constrained, and at this time, the information between the sampling sub-image features of different categories is also represented, and the computer device can obtain the block sparse matrix 602 and constrain it, so as to obtain the block sparse global self-attention weight matrix 603 after constraint, and as can be seen from FIG. 6, in the block sparse global self-attention weight matrix 603, only the information between the sampling sub-image features belonging to the same category is represented. The block sparse constraint essentially utilizes the association of categories to screen the global self-attention, and only the attention of the same category is focused on and retained, and the attention of different categories is removed. Compared to the global self-attention weight matrix 601, the block sparse global self-attention weight matrix 603 utilizes the category information of the instance to learn attention, and performs appropriate simplified calculation.

[0066] Step S208: The self-attention network layer determines block sparse self-attention corresponding to each sampling sub-image feature based on the at least two classification clusters and the block sparse global self-attention weight matrix.

[0067] Specifically, the second attention sub-network further includes a value weight matrix, and the N sampled sub-image features are represented as sampled sub-image feature N aThe computer device multiplies the sampling sub-image feature matrix and the value weight matrix through a self-attention network layer to obtain a value matrix, and calculates the sampling sub-image feature N a The sampling sub-image feature in the classification cluster to which N belongs is defined as the target sampling sub-image feature, and the sampling sub-image feature N a and the target sampling sub-image feature as the target block sparse global self-attention weight; obtain a value vector corresponding to the target sampling sub-image feature from the value matrix as a target value vector; and calculate the sampling sub-image feature N based on the target value vector and the target block sparse global self-attention weight. a where we determine the block sparse self-attention corresponding to the sampling sub-image feature N a is the above sampling sub-image feature matrix

number

[0068] Specifically, the calculation of the value matrix can be expressed by equation (8).

number

number

number

[0069] The calculation of block sparse self-attention can be expressed by equation (9).

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0070] Step S209: The average value pooling network layer performs an average value pooling process on at least two of the block sparse self-attentions to obtain a second feature vector.

[0071] Specifically, the average value pooling process refers to adding at least two block sparse self-attentions and then taking the average to obtain the vector, i.e., the second feature vector X2.

[0072] Step S210: The first feature vector and the second feature vector are input into a classification sub-network of an image recognition model, and the classification sub-network includes a feature fusion network layer and a classification network layer, and the feature fusion network layer performs feature fusion processing on the first feature vector and the second feature vector to obtain a fusion feature vector, and the classification network layer performs classification processing on the fusion feature vector to obtain a classification result of the image to be detected.

[0073] Specifically, the first attention sub-network outputs a first feature vector X1, the second attention sub-network outputs a second feature vector X2, and the two parallel feature vectors perform feature fusion in the feature fusion network layer, and the classification network layer can adopt an MLP classifier, so that the final output can be expressed as the following equation (10).

number

[0074] In order to facilitate understanding of the structure of the image identification model in the above steps S202 to S210, please also refer to FIG. 7, which is a structural schematic diagram of the image identification model provided by the embodiment of the present application. As shown in FIG. 7, the image identification model 7 includes a first attention sub-network 71, a second attention sub-network 72, and a classification sub-network 73. After obtaining a feature representation subset 700 corresponding to the image to be detected, the computer device sequentially inputs the sub-image features in the feature representation subset 700 to the image identification model 7. Here, the feature representation subset 700 includes at least two sub-image features. In the image identification model 7, the computer device inputs the feature representation subset 700 to the first attention sub-network 71 and the second attention sub-network 72, respectively. In the first attention sub-network 71, for each sub-image feature in the feature representation subset 700, a corresponding weight is learned by a parameterized neural network, for example, the sub-image feature 711 may be input to the parameterized neural network 712, and the neural network 712 outputs the weight of the sub-image feature 711, and the specific implementation can refer to the above step S203. Then, in the first attention sub-network 71, the computer device adopts a nonlinear attention weighting method of linear terms to perform effective aggregation for every sub-image feature in the feature representation subset 700, and finally obtains a first feature vector 713, where the aggregation process can refer to the above step S204. In the second attention sub-network 72, the computer device first performs unsupervised clustering on the feature representation subset 700 to obtain at least two classification clusters, for example, classification cluster 721, and the sub-images corresponding to the sub-image features in the classification cluster 721 belong to the same category of images. Then, the computer device obtains some sub-image features from each classification cluster as sampling sub-image features, where the clustering and sampling process can refer to the above step S206.Next, in the second attention sub-network 72, the computer device can obtain a key matrix 723, a query matrix 724, and a value matrix 725 by matrix transforming the sampling sub-image feature matrix 722 composed of the sampling sub-image features, where the matrix transformation can refer to the above formula (5), formula (6), and formula (8), and specifically can be realized by a convolution network with a convolution kernel of 1×1. Next, in the second attention sub-network 72, a block sparse global self-attention matrix 726 can be determined based on the transposed matrix of the key matrix 723, the query matrix 724, and the block sparse matrix, and the determining process can refer to the above step S207. Furthermore, a second feature vector 727 is determined based on the block sparse global attention matrix 726 and the value matrix 725, and the determining process can refer to the above step S208 and step S209. Finally, the computer device inputs the first feature vector 713 and the second feature vector 727 into the classification sub-network 73. In the classification sub-network 73, after performing feature fusion on the first feature vector 713 and the second feature vector 727, the feature fused vector is input to a classifier 731, which then outputs a classification result 732, which may include an image normal probability and an image abnormal probability.

[0075] By adopting the method provided in the embodiments of the present application, the first attention sub-network and the second attention sub-network in the image recognition model use two different methods to mine information of the image to be detected, and obtain a first feature vector and a second feature vector. After being fused, the two feature vectors complement each other and constrain each other, and the feature vector after fusion can be predicted, and the obtained classification result has high accuracy.

[0076] Further, as shown in FIG. 8, FIG. 8 is a flow diagram of an initial image recognition model training method provided by an embodiment of the present application. Here, the initial image recognition model training method may be executed by a computer device, where the computer device may be the service server 100 shown in FIG. 1a above, or any terminal device in the terminal device group shown in FIG. 1a above, for example, the terminal device 10c. Hereinafter, the initial image recognition model training method will be described as being executed by a computer device as an example. Here, the initial image recognition model training method may include at least the following steps S301 to S305.

[0077] Step S301: Obtain a sample image, perform a feature extraction process on the sample image, and obtain a sample feature representation subset of the sample image, where the sample image includes at least two sample sub-images, the sample feature representation subset includes at least two sample sub-image features, and the at least two sample sub-image features have a one-to-one correspondence with the at least two sample sub-images.

[0078] Specifically, for the implementation process of step S301, reference may be made to the description of step S101 in the embodiment corresponding to FIG. 2 above, and detailed description thereof will be omitted here.

[0079] Step S302: Input the at least two sample sub-images into an initial image identification model, generate sample attention weights corresponding to each of the at least two sample sub-image features using the initial image identification model, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector.

[0080] Specifically, the initial image classification model may include a first initial attention sub-network, and the computer equipment can use the first initial attention sub-network to generate sample attention weights corresponding to each of the at least two sample sub-image features, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector. For the specific implementation process, please refer to the description of steps S202 to S204 in the embodiment corresponding to Figure 4 above, and detailed explanations will be omitted here.

[0081] Step S303: Using the initial image identification model, perform a clustering sampling process on the at least two sample sub-image features to obtain at least two sample classification clusters, where the sample classification clusters include sample sampling sub-image features; determine sample block sparse self-attention corresponding to each of the sample sampling sub-image features based on the at least two sample classification clusters and a block sparse matrix; determine a second sample feature vector based on the at least two sample block sparse self-attentions, and the sample block sparse self-attention corresponding to the sample sampling sub-image feature is determined based on the sample sampling sub-image feature in the sample classification cluster to which it belongs.

[0082] Specifically, the initial image classification model may further include a second initial attention sub-network, and then the second initial attention sub-network performs a clustering sampling process on at least two sample sub-image features to obtain sample sampling sub-image features contained in each of the at least two sample classification clusters, determines sample block sparse self-attention corresponding to each sample sampling sub-image feature based on the at least two sample classification clusters and the block sparse matrix, and determines a second sample feature vector based on the at least two sample block sparse self-attention. For the specific implementation process, please refer to the description of steps S205 to S209 in the embodiment corresponding to Figure 4 above, and detailed explanations will be omitted here.

[0083] Step S304: Determine a sample classification result of the sample image according to the first sample feature vector and the second sample feature vector through the initial image identification model.

[0084] Specifically, the initial image identification model may further include an initial classification sub-network, and then the initial classification sub-network determines a sample classification result of the sample image based on the first sample feature vector and the second sample feature vector. For specific implementation, please refer to the description of step S210 in the embodiment corresponding to Figure 4 above, and detailed explanation will be omitted here.

[0085] Step S305: Based on the at least two sample classification clusters, the attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and the classification labels corresponding to the sample images, perform model parameter adjustment for the initial image classification model to obtain an image classification model used to identify the classification results of the image to be detected.

[0086] Specifically, the inputs of the first attention sub-network and the second attention sub-network in the finally obtained image discrimination model are both the same sub-image features, so the attention distribution of the first attention sub-network to at least two sub-image features should be consistent with the attention distribution of the second attention sub-network to at least two sub-image features, so that in the process of training the initial image discrimination model, the computer device can first determine a divergence loss value according to the sample attention weights corresponding to at least two sample classification clusters and at least two sample sub-image features, and then determine a classification loss value according to the sample classification result and the classification label corresponding to the sample image, and finally calculate a weighted sum of the divergence loss value and the classification loss value to obtain a model total loss value, and perform model parameter adjustment for the initial image discrimination model according to the model total loss value to obtain an image discrimination model. Here, the divergence loss value is used to ensure that the attention distributions input to similar sub-image features by the two network branches of the finally obtained image discrimination model are consistent. The classification loss value is used to ensure that the classification result output by the final trained image recognition model can be close to the actual result.

[0087] Specifically, the above implementation process of determining the divergence loss value according to at least two sample classification clusters and sample attention weights corresponding to each of the at least two sample sub-image features may be as follows: obtain an i-th sample classification cluster in the at least two sample classification clusters, where i is a positive integer and is less than or equal to the quantity of the at least two sample classification clusters, take the sample sub-image feature included in the i-th sample classification cluster as a target sample sub-image feature, and determine a category divergence loss value corresponding to the i-th sample classification cluster according to the sample attention weights corresponding to the target sample sub-image feature and the quantity of the target sample sub-image feature, and add the category divergence loss values ​​corresponding to each sample classification cluster to obtain a divergence loss value.

[0088] When the computer device performs image detection on the sample image, it performs clustering on the sample sub-image features contained in the sample image in the second initial attention sub-network to obtain at least two sample classification clusters, and since the attention degrees of the sample sub-image features in the common sample classification clusters in the second initial attention sub-network are the same, the attention degrees of the sample sub-image features in the common sample classification clusters in the first initial attention sub-network should also be the same. For example, a sample image includes six sample sub-image features, namely B1, B2, B3, B4, B5, and B6, the sample attention weights generated in the first initial attention sub-network are 0.10, 0.22, 0.11, 0.31, 0.22, 0.12, respectively, and the sample classification clusters generated in the second initial attention sub-network are sample classification cluster 1 {B1, B3, B6}, sample classification cluster 2 {B2, B4, B5}. Thus, the sample attention weights corresponding to B1, B3, and B4 in sample classification cluster 1 are close to the matching state and are reasonable; however, the sample attention weight corresponding to B4 in sample classification cluster 2 is significantly higher than B2 and B5, which is not reasonable and therefore needs to be adjusted by the divergence loss value. That is, the attention weights generated in the first attention sub-network by the sample sub-image features in the common sample classification cluster can follow a uniform distribution, so that each sample classification cluster can determine one category divergence loss value. Finally, the category divergence loss values ​​corresponding to each sample classification cluster are added to obtain a divergence loss value.

[0089] Specifically, the implementation process of determining a category divergence loss value corresponding to the i-th sample classification cluster based on the sample attention weights corresponding to the target sample sub-image features and the quantity of the target sample sub-image features may be as follows: obtain a fitting attention distribution composed of sample attention weights corresponding to the target sample sub-image features, perform a normalization process on the fitting attention weight distribution to obtain a normalized fitting attention distribution, and take the uniform attention distribution corresponding to the quantity of the target sample sub-image features as the attention distribution label, and determine a category divergence loss value corresponding to the i-th sample classification cluster based on the normalized fitting attention distribution and the attention distribution label.

[0090] Suppose the sample attention weights corresponding to the target sample sub-image features are 0.10, 0.12, 0.11, the constructed fitting attention distribution is, i.e., [0.10, 0.12, 0.11], in order to facilitate the calculation of the subsequent category divergence loss value, the input is required to be a probability, so a normalization process is performed on the fitting attention distribution, i.e., the sum by addition needs to be 1, and the obtained normalized fitting attention distribution is [0.303, 0.363, 0.333]. When the quantity of the target sample sub-image features is 3, the corresponding uniform attention distribution is the attention distribution label, which is [1 / 3, 1 / 3, 1 / 3].

[0091] The process of determining the category divergence loss value corresponding to the i-th sample classification cluster based on the normalized fitted attention distribution and the attention distribution label can be expressed by the following equation (11).

number

number

number

number

[0092] Therefore, the calculation of the dissipation loss value can be realized by the following equation (12).

number

number

[0093] Specifically, the total loss value can be determined by the following equation (13).

number

[0094] Specifically, when training the initial image recognition model, it can be trained for 100 epochs (periods), the optimizer adopts Adam (optimization algorithm) by default, the initialization learning rate is 1e-4, a cosine annealing strategy is adopted to adjust the learning rate, and the minimum learning rate is 1e-6.

[0095] By adopting the method provided in the present embodiment, when training the initial image recognition model, additional constraints are placed on the attention distribution of the first initial attention sub-network and the second initial attention sub-network, i.e., the KL divergence loss function is increased in the loss function, ensuring that the attention distributions of the two sub-networks input for similar multi-instances are consistent, and the image detection accuracy of the final trained image recognition model is high.

[0096] Referring to FIG. 9, FIG. 9 is a structural schematic diagram of an image detection device provided in the embodiment of the present application. The image detection device may be a computer program (including program code) that operates a computer device, for example, the image detection device is an application software, and the device can be used to perform appropriate steps in the image detection method provided in the embodiment of the present application. As shown in FIG. 9, the image detection device 1 may include a feature extraction module 11, a first vector generation module 12, a second vector generation module 13, and a classification module 14.

[0097] The feature extraction module 11 is used for obtaining an image of a detection target, performing a feature extraction process on the image of the detection target, and obtaining a feature representation subset of the image of the detection target, where the image of the detection target includes at least two sub-images, the feature representation subset includes at least two sub-image features, and the at least two sub-image features have a one-to-one correspondence with the at least two sub-images; The first vector generating module 12 is used for generating attention weights corresponding to at least two sub-image features, and performing weighted aggregation processing on the at least two sub-image features according to the attention weights to obtain a first feature vector; The second vector generation module 13 performs a clustering sampling process on at least two sub-image features to obtain at least two classification clusters, the classification clusters including the sampled sub-image features, and determines block sparse self-attention corresponding to each sampled sub-image feature according to the at least two classification clusters and the block sparse matrix, and is used to determine a second feature vector based on the at least two block sparse self-attentions, where the block sparse self-attention corresponding to the sampled sub-image feature is determined according to the sampled sub-image feature in the classification cluster to which it belongs; The classification module 14 is used for determining a classification result of the target image based on the first feature vector and the second feature vector.

[0098] Here, for the specific implementation methods of the feature extraction module 11, the first vector generation module 12, the second vector generation module 13, and the classification module 14, please refer to the relevant description of the embodiment corresponding to Figure 2 above, and detailed explanations will be omitted here.

[0099] Here, the feature extraction module 11 includes a pre-processing unit 111 and a feature extraction unit 112 .

[0100] The pre-processing unit 111 is used to distinguish between background and foreground areas in the image to be detected; The pre-processing unit 111 is further used for performing image segmentation on the image of the detection target according to the background region and the foreground region to obtain a foreground image of the detection target; The pre-processing unit 111 is further used for performing a scaling process on the foreground image to be detected based on the scaling ratio to obtain a foreground image to be trimmed; The pre-processing unit 111 is further used for performing a cropping process on the foreground image to be cropped according to the preset length of the sub-image and the preset width of the sub-image to obtain at least two sub-images, where the preset length of the sub-image is less than the length of the foreground image to be cropped, and the preset width of the sub-image is less than the width of the foreground image to be cropped; The feature extraction unit 112 is used to perform image feature extraction processing on at least two sub-images respectively, obtain sub-image features corresponding to each of the at least two sub-images, and determine a feature representation subset of the image to be detected based on the sub-image features corresponding to each of the at least two sub-images.

[0101] Here, for the specific implementation of the pre-processing unit 111 and the feature extraction unit 112, reference may be made to the relevant description of the embodiment corresponding to FIG. 2 above, and detailed description thereof will be omitted here.

[0102] Here, the first vector generating module 12 includes a first input unit 121, a weight fitting unit 122, and an aggregation unit 123.

[0103] The first input unit 121 is used for inputting at least two sub-image features into a first attention sub-network in the image discrimination model, the first attention sub-network including a weight learning network layer and a weight summation network layer; The weight fitting unit 122 is used for performing weight fitting processing on the at least two sub-image features through a weight learning network layer respectively to obtain attention weights corresponding to the at least two sub-image features; The aggregation unit 123 is used to perform weighting processing on each sub-image feature based on the attention weight through a weighted aggregation network layer, obtain weighted sub-image features corresponding to each sub-image feature, and perform aggregation processing on the weighted sub-image features corresponding to each of the at least two sub-image features to obtain a first feature vector.

[0104] Here, for specific implementations of the first input unit 121, the weight fitting unit 122, and the aggregation unit 123, reference may be made to the relevant description of the embodiment corresponding to FIG. 4 above, and detailed description thereof will be omitted here.

[0105] Here, the second vector generating module 13 includes a second input unit 131, a clustering sampling unit 132, a global self-attention determination unit 133, a self-attention determination unit 134, and an average value pooling unit 135.

[0106] The second input unit 131 is used for inputting at least two sub-image features into a second attention sub-network in the image discrimination model, the second attention sub-network including: a clustering sampling network layer, a global self-attention network layer, a self-attention network layer, and an average pooling network layer; The clustering sampling unit 132 is used for performing clustering sampling processing on the at least two sub-image features through a clustering sampling network layer to obtain at least two classification clusters including the sampling sub-image features, and the sum of the quantity of the sampling sub-image features included in each of the at least two classification clusters is N, where N is a positive integer less than the quantity of the at least two sub-image features; The global self-attention determination unit 133 is used for determining a block sparse global self-attention weight matrix of the N sampling sub-image features according to the block sparse matrix by the global self-attention network layer; The self-attention determination unit 134 is used for determining, by a self-attention network layer, block sparse self-attention corresponding to each sampling sub-image feature according to at least two classification clusters and the block sparse global self-attention weight matrix; The average value pooling unit 135 is used for performing an average value pooling process on the at least two block sparse self-attentions through an average value pooling network layer to obtain a second feature vector.

[0107] Here, for the specific implementation methods of the second input unit 131, the clustering sampling unit 132, the global self-attention determination unit 133, the self-attention determination unit 134, and the average value pooling unit 135, please refer to the relevant description of the embodiment corresponding to Figure 4 above, and detailed description will be omitted here.

[0108] Here, the clustering sampling unit 132 includes a clustering subunit 1321 and a sampling subunit 1322 .

[0109] The clustering subunit 1321 is used for performing clustering processing on at least two sub-image features through a clustering sampling network layer to obtain at least two classification clusters; The sampling subunit 1322 is used for obtaining a k-th classification cluster in the at least two classification clusters, where k is a positive integer, and the k-th classification cluster includes at least one clustering sub-image feature; The sampling subunit 1322 is further used for obtaining a vector distance between each of the at least one clustering sub-image features and the cluster center of the k-th classification cluster as a reference distance; The sampling subunit 1322 is further used for sequentially obtaining h clustering sub-image features in at least one clustering sub-image feature based on the reference distance, and setting the h clustering sub-image features as sampling sub-image features included in the kth classification cluster, where h is a positive integer, and h is less than or equal to the quantity of the at least one clustering sub-image feature.

[0110] Here, for the specific implementation manner of the clustering subunit 1321 and the sampling subunit 1322, reference may be made to the relevant description of the embodiment corresponding to FIG. 4 above, and detailed description thereof will be omitted here.

[0111] Here, the second attention sub-network includes a query weight matrix and a key weight matrix.

[0112] The global self-attention determination unit 133 includes a first initialization subunit 1331 , a matrix determination subunit 1332 , and a normalization subunit 1333 .

[0113] The first initialization subunit 1331 is used for constructing a sampling sub-image feature matrix according to the sampling sub-image features contained in each of the at least two classification clusters by a global self-attention network layer; The first initialization sub-unit 1331 is further used for multiplying the sampling sub-image feature matrix and the query weight matrix to obtain a query matrix; and multiplying the sampling sub-image feature matrix and the key weight matrix to obtain a key matrix; The matrix determination subunit 1332 is used for determining a block sparse global relevance matrix according to the query matrix, the transpose matrix corresponding to the key matrix, and the block sparse matrix; The normalization sub-unit 1333 is used to perform a normalization process on the block sparse global relevance matrix to obtain a block sparse global self-attention weight matrix.

[0114] Here, for the specific implementation methods of the first initialization subunit 1331, the matrix determination subunit 1332, and the normalization subunit 1333, reference may be made to the relevant description of the embodiment corresponding to Figure 4 above, and detailed description here will be omitted.

[0115] Here, the second attention sub-network further includes a value weight matrix, and the N sampled sub-image features are represented as sampled sub-image feature Nj where j is a positive integer less than or equal to N, The self-attention determination unit 134 includes a target acquisition subunit 1341 and a determination subunit 1342 .

[0116] The target acquisition subunit 1341 is used to multiply the sampling sub-image feature matrix and the value weight matrix by the self-attention network layer to obtain a value matrix; The target acquisition subunit 1341 further includes sampling sub-image feature N j The sampling sub-image feature in the classification cluster to which the target image belongs is used as the target sampling sub-image feature; The target acquisition subunit 1341 further selects the sampled sub-image feature N from the block sparse global self-attention weight matrix. j and the block sparse global self-attention weight between the target sampling sub-image feature is obtained as the target block sparse global self-attention weight; The target acquisition subunit 1341 is further used for acquiring a value vector corresponding to the target sampling sub-image feature from the value matrix as a target value vector; The determining subunit 1342 determines the sampling sub-image feature N based on the target value vector and the target block sparse global self-attention weights. j The block sparse self-attention corresponding to

[0117] Here, for specific implementations of the target acquisition subunit 1341 and the determination subunit 1342, reference may be made to the relevant description of the embodiment corresponding to FIG. 4 above, and detailed description thereof will be omitted here.

[0118] Here, the classification module 14 includes a third input unit 141, a feature fusion unit 142, and a classification unit 143.

[0119] The third input unit 141 is used for inputting the first feature vector and the second feature vector into a classification sub-network of an image recognition model, the classification sub-network including a feature fusion network layer and a classification network layer; The feature fusion unit 142 is used to perform feature fusion processing on the first feature vector and the second feature vector through a feature fusion network layer to obtain a fusion feature vector; The classification unit 143 is used to perform classification processing on the fusion feature vector through a classification network layer to obtain a classification result of the image to be detected.

[0120] Here, for the specific implementation manners of the third input unit 141, the feature fusion unit 142, and the classification unit 143, please refer to the relevant description of the embodiment corresponding to Figure 4 above, and detailed description will be omitted here.

[0121] Referring to FIG. 10, FIG. 10 is a structural schematic diagram of a computer device provided by an embodiment of the present application. As shown in FIG. 10, the image detection device 1 in the embodiment corresponding to FIG. 10 can be applied to a computer device 1000, which may include a processor 1001, a network interface 1004, and a memory 1005, and the computer device 1000 may further include a user interface 1003 and at least one communication bus 1002. Here, the communication bus 1002 is used to realize the connection communication between these assemblies. Here, the user interface 1003 may include a display screen (Display) and a keyboard (Keyboard), and optionally, the user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one magnetic disk memory. The memory 1005 may optionally be at least one storage device remote from the processor 1001. As shown in FIG. 10, the memory 1005, which serves as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0122] In the computer equipment 1000 shown in FIG. 10, the network interface 1004 can provide a network communication network element, the user interface 1003 is mainly used to provide an input interface for a user, and the processor 1001 is used to call the equipment control application program stored in the memory 1005, thereby realizing the image detection method provided by the embodiment of the present application.

[0123] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the description of the image detection method in the embodiment corresponding to either of Figures 2 and 4 in the preceding paragraph, and detailed description thereof will be omitted here, and detailed description of the beneficial effects of adopting the same method will also be omitted.

[0124] It should also be noted that the present embodiment further provides a computer-readable storage medium, in which a computer program is stored for the image detection device 1 mentioned above, the computer program including program instructions, and the processor can execute the image detection method described in the embodiment corresponding to either FIG. 2 or FIG. 4 above when executing the program instructions, and therefore a detailed description is omitted here. Also, a detailed description of the beneficial effects of the same method is omitted. For technical details not disclosed in the embodiment of the computer-readable storage medium related to the present application, please refer to the description of the method embodiment of the present application.

[0125] Further, refer to FIG. 11, which is a structural schematic diagram of another image detection device provided by the embodiment of the present application. The image detection device 2 may be a computer program (including program code) running in a computer device, for example, the image detection device 2 is an application software, and the device can be used to perform the processing steps in the method provided by the embodiment of the present application. As shown in FIG. 11, the image detection device 2 may include a sample feature extraction module 21, a first sample vector generation module 22, a second sample vector generation module 23, a sample classification module 24, and a training module 25.

[0126] The sample feature extraction module 21 is used for obtaining a sample image, performing feature extraction processing on the sample image, and obtaining a sample feature representation subset of the sample image, where the sample image includes at least two sample sub-images, the sample feature representation subset includes at least two sample sub-image features, and the at least two sample sub-image features have a one-to-one correspondence with the at least two sample sub-images; The first sample vector generation module 22 is used for inputting at least two sample sub-images into an initial image discrimination model, generating sample attention weights corresponding to the at least two sample sub-image features through the initial image discrimination model, and performing a weighted aggregation process on the at least two sample sub-image features according to the sample attention weights corresponding to the at least two sample sub-image features to obtain a first sample feature vector; The second sample vector generation module 23 performs a clustering sampling process on the at least two sample sub-image features according to the initial image discrimination model to obtain at least two sample classification clusters, the sample classification clusters including the sample sampling sub-image features, and determines sample block sparse self-attention corresponding to each sample sampling sub-image feature according to the at least two sample classification clusters and the block sparse matrix, which is used for determining a second sample feature vector according to the at least two sample block sparse self-attentions, and the sample block sparse self-attention corresponding to the sample sampling sub-image feature is determined according to the sample sampling sub-image feature in the sample classification cluster to which it belongs; The sample classification module 24 is used to determine a sample classification result of the sample image according to the first sample feature vector and the second sample feature vector according to the initial image identification model; The training module 25 is used to adjust model parameters for the initial image identification model based on at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and the classification labels corresponding to the sample images, to obtain an image identification model used to identify the classification results of the image to be detected.

[0127] Here, for specific implementation methods of the sample feature extraction module 21, the first sample vector generation module 22, the second sample vector generation module 23, the sample classification module 24, and the training module 25, please refer to the relevant description of the embodiment corresponding to Figure 8 above, and detailed description will be omitted here.

[0128] Here, the training module 25 includes a divergence loss value determining unit 251, a classification loss value determining unit 252, a weighted sum obtaining unit 253, and a model adjusting unit 254.

[0129] The divergence loss value determining unit 251 is used for determining a divergence loss value according to the at least two sample classification clusters and the sample attention weights corresponding to each of the at least two sample sub-image features; The classification loss value determination unit 252 is used to determine a classification loss value according to the sample classification result and the classification label corresponding to the sample image; The weighted sum obtaining unit 253 is used for obtaining a weighted sum of the divergence loss value and the classification loss value to obtain a model total loss value; The model adjustment unit 254 is used to perform model parameter adjustment on the initial image recognition model based on the model total loss value to obtain an image recognition model. Here, for specific implementation methods of the divergence loss value determination unit 251, the classification loss value determination unit 252, the weighted sum acquisition unit 253, and the model adjustment unit 254, please refer to the relevant description of the embodiment corresponding to Figure 8 above, and detailed description will be omitted here.

[0130] Here, the divergence loss value determining unit 251 includes an acquisition subunit 2511 , a category loss value determining subunit 2512 , and a total loss value determining subunit 2513 .

[0131] The acquiring subunit 2511 is used for acquiring an i-th sample classification cluster in the at least two sample classification clusters, where i is a positive integer, and i is less than or equal to the quantity of the at least two sample classification clusters; The obtaining subunit 2511 is further used for determining the sample sub-image features included in the i-th sample classification cluster as target sample sub-image features; The category loss value determining subunit 2512 is used for determining a category divergence loss value corresponding to the i-th sample classification cluster according to the sample attention weights corresponding to the target sample sub-image features and the quantity of the target sample sub-image features; The total loss value determination subunit 2513 sums the category divergence loss values ​​corresponding to each sample classification cluster to obtain a divergence loss value.

[0132] Here, the category loss value determination subunit 2512 is specifically used for obtaining a fitting attention distribution composed of sample attention weights corresponding to target sample sub-image features, performing a normalization process on the fitting attention weight distribution to obtain a normalized fitting attention distribution, taking the uniform attention distribution corresponding to the quantity of the target sample sub-image features as the attention distribution label, and determining a category divergence loss value corresponding to the i-th sample classification cluster based on the normalized fitting attention distribution and the attention distribution label.

[0133] Here, for the specific implementation manners of the acquisition subunit 2511, the category loss value determination subunit 2512, and the total loss value determination subunit 2513, reference may be made to the relevant description of the embodiment corresponding to Figure 8 above, and detailed description here will be omitted.

[0134] Further, referring to FIG. 12, FIG. 12 is a structural schematic diagram of another computer device provided by the embodiment of the present application. As shown in FIG. 12, the image detection device 2 in the embodiment corresponding to FIG. 11 above can be applied to a computer device 2000, which may include a processor 2001, a network interface 2004, and a memory 2005, and the computer device 2000 further includes a user interface 2003 and at least one communication bus 2002. Here, the communication bus 2002 is used to realize the connection communication between these assemblies. Here, the user interface 2003 may include a display screen (Display) and a keyboard (Keyboard), and optionally, the user interface 2003 may further include a standard wired interface, a wireless interface. The network interface 2004 may optionally include a standard wired interface, a wireless interface (e.g., a WI-FI interface). The memory 2005 may be a high-speed RAM memory or a non-volatile memory, such as at least one magnetic disk memory. The memory 2005 may optionally be at least one storage device remote from the processor 2001. As shown in FIG. 12, the memory 2005, which serves as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0135] In the computer equipment 2000 shown in FIG. 12, the network interface 2004 can provide network communication functions, the user interface 2003 is mainly used to provide an input interface for a user, and the processor 2001 can be used to call the equipment control application program stored in the memory 2005, thereby realizing the training method for the initial image recognition model provided in the embodiments of the present application.

[0136] It should be understood that the computer device 2000 described in the embodiment of the present application can execute the description of the access control method in each embodiment above, and can also execute the description of the image detection device 2 in the embodiment corresponding to Fig. 11 above, and detailed description here will be omitted. Also, detailed description of the beneficial effects of adopting the same method will be omitted.

[0137] It should also be noted that the present embodiment further provides a computer-readable storage medium, in which a computer program is stored for execution by the image detection device 2 mentioned above, and the processor can execute the description of the access control method of any of the embodiments above when loading and executing the computer program, and therefore a detailed description is omitted here. Also, a detailed description of the beneficial effects of adopting the same method is omitted. For technical details not disclosed in the embodiments of the computer-readable storage medium related to the present application, please refer to the description of the method embodiments of the present application.

[0138] The computer-readable storage medium may be the image detection device provided by any of the above embodiments, or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may be an external storage device of the computer device, such as a removable hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., arranged on the computer device. In addition, the computer-readable storage medium may not only include the internal storage unit of the computer device, but also includes an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0139] It should also be noted that the present embodiment further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computer device reading the computer instructions from the computer readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the method provided by any of the embodiments corresponding to the preceding sentence.

[0140] The terms "first", "second", etc. in the specification, claims, and drawings of the present embodiment are used to distinguish different objects, not to describe a specific order. Also, the term "comprises" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the steps or modules already listed, and may optionally further include steps or modules not listed, or may optionally further include other step units specific to the process, method, apparatus, product, or device.

[0141] As those skilled in the art can recognize, the units and algorithm steps of each example described in combination with the embodiments disclosed herein may be realized in electronic hardware, computer software, or a combination of both, and in order to clearly explain the compatibility of hardware and software, the above description has already generally described the configurations and steps of each example according to network elements. Whether those network elements are implemented in a hardware or software manner is determined by the specific application and design constraints of the technical solution. Those skilled in the art may implement the described network elements using different methods for each specific application, and such implementation should not be considered to go beyond the scope of this application.

[0142] What has been presented above is merely a preferred embodiment of the present application, which of course cannot limit the scope of the present application, and therefore equivalent changes made according to the claims of the present application still belong to the scope covered by the present application. [Explanation of symbols]

[0143] 10 Terminal Equipment 11 Feature Extraction Module 12 First Vector Generation Module 13 Second Vector Generation Module 14 Classification Module 21 Sample Feature Extraction Module 22 First sample vector generation module 23 Second sample vector generation module 24 Sample Classification Modules 25 Training Modules 50 Feature Extractor RestNet 71 First Attention Sub-Network 72 Second Attention Sub-Network 73 Classification Subnetwork 100 Service Server 111 Pretreatment unit 112 Feature Extraction Unit 121 First input unit 122 Fitting unit 123 tally units 131 Second input unit 132 Clustering Sampling Units 133 Global Self-Attention Decision Unit 134 Self-Attention Decision Unit 135 Mean Pooling Units 141 3rd input unit 142 Feature Fusion Unit 143 Classification Units 200 Terminal Equipment 251 Divergence Loss Value Determination Unit 252 Classification Loss Value Determination Unit 253 Weighted Sum Acquisition Units 254 Model Adjustment Unit 300 Patient Management Applications 301 Colorectal Pathology Images 400 Service Server 401 Feature Representation Subset 402 Image Recognition Model 402 Training Image Recognition Model 403 First feature vector 404 Second feature vector 405 Classification results 601 Queue 602 Block Sparse Matrix 603 Queue 700 feature representation subset 712 Neural Networks 713 First feature vector 721 Classification Clusters 722 Sampling Sub-Image Feature Matrix 723 Key Matrix 724 Queue 725 Value Matrix 726 Block Sparse Global Self-Attention Matrix 726 Block Sparse Global Attention Matrix 727 Second feature vector 731 Classifier 732 Classification results 1000 Computer Equipment 1001 Processor 1002 Communication Bus 1003 User Interface 1004 Network Interface 1005 Memory 1321 Clustering Subunit 1322 Sampling Subunit 1331 First Initialization Subunit 1332 Matrix Determination Subunit 1333 Normalization Subunit 1341 Target Acquisition Subunit 1342 Decision Subunit 2000 Computer Equipment 2001 Processor 2002 Communication Bus 2003 User Interface 2004 Network Interface 2005 Memory 2511 Acquisition Subunit 2512 Category Loss Value Determination Subunit 2513 Total loss value determination subunit 3000 images 3001 Foreground area 3002 Background area 3003 Foreground Image 3004 Sub-image 3005 Sub-Image 3006 Sub-Image 3007 Feature Extractor 3008 Feature Representation Subset 4021 First Attention Sub-Network 4022 Second Attention Sub-Network 4023 Classification Subnetwork

Claims

1. 1. A method of image detection implemented by a computing device, comprising: acquiring an image of the object to be detected, and performing a feature extraction process on the image of the object to obtain a feature representation subset of the image of the object to be detected, the image of the object to be detected including at least two sub-images, the feature representation subset including at least two sub-image features, the at least two sub-image features having a one-to-one correspondence with the at least two sub-images; generating an attention weight corresponding to each of the at least two sub-image features, and performing a weighted aggregation process on the at least two sub-image features based on the attention weight to obtain a first feature vector; performing a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, the classification clusters including the sampled sub-image features; determining block sparse self-attention corresponding to each of the sampled sub-image features based on the at least two classification clusters and a block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions, where the block sparse self-attention corresponding to the sampled sub-image feature is determined based on the sampled sub-image feature in the classification cluster to which it belongs; determining a classification result of the image to be detected based on the first feature vector and the second feature vector.

2. The step of performing a feature extraction process on the image of the detection target to obtain a feature representation subset of the image of the detection target includes: identifying background and foreground regions in the image of the detection target; performing image segmentation on the image of the detection target based on the background region and the foreground region to obtain a foreground image of the detection target; performing a scaling process on the foreground image to be detected based on a scaling factor to obtain a foreground image to be trimmed; performing a cropping process on the foreground image to be cropped based on a preset length of a sub-image and a preset width of a sub-image to obtain the at least two sub-images, wherein the preset length of the sub-image is less than the length of the foreground image to be cropped, and the preset width of the sub-image is less than the width of the foreground image to be cropped; 2. The method of claim 1, further comprising: performing an image feature extraction process on each of the at least two sub-images to obtain sub-image features corresponding to each of the at least two sub-images; and determining a feature representation subset of the image to be detected based on the sub-image features corresponding to each of the at least two sub-images.

3. generating attention weights corresponding to the at least two sub-image features, and performing a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; inputting the at least two sub-image features into a first attention sub-network in an image discrimination model, the first attention sub-network including a weight learning network layer and a weight aggregation network layer; performing a weight fitting process on the at least two sub-image features by the weight learning network layer to obtain attention weights corresponding to the at least two sub-image features; 2. The method of claim 1 , further comprising: weighting each of the sub-image features based on the attention weights by the weighted aggregation network layer to obtain a weighted sub-image feature corresponding to each of the sub-image features; and aggregating the weighted sub-image features corresponding to each of the at least two sub-image features to obtain the first feature vector.

4. performing a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, the classification clusters including the sampled sub-image features; determining block sparse self-attentions corresponding to each of the sampled sub-image features based on the at least two classification clusters and a block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions; inputting the at least two sub-image features into a second attention sub-network in an image discrimination model, the second attention sub-network including a clustering sampling network layer, a global self-attention network layer, a self-attention network layer, and an average pooling network layer; performing a clustering sampling process on the at least two sub-image features by the clustering sampling network layer to obtain the at least two classification clusters, the classification clusters including the sampling sub-image features, a sum of the quantity of the sampling sub-image features included in each of the at least two classification clusters is N, where N is a positive integer less than the quantity of the at least two sub-image features; determining a block-sparse global self-attention weight matrix for the N sampled sub-image features based on the block-sparse matrix by the global self-attention network layer; determining, by the self-attention network layer, block sparse self-attention corresponding to each of the sampled sub-image features based on the at least two classification clusters and the block sparse global self-attention weight matrix; and performing an average pooling process on at least two of the block sparse self-attention vectors by the average pooling network layer to obtain a second feature vector.

5. The step of performing a clustering sampling process on the at least two sub-image features by the clustering sampling network layer to obtain the at least two classification clusters includes: performing a clustering process on the at least two sub-image features by the clustering sampling network layer to obtain the at least two classification clusters; obtaining a k-th classification cluster in the at least two classification clusters, where k is a positive integer, and the k-th classification cluster includes at least one clustering sub-image feature; obtaining a vector distance between each of the at least one clustering sub-image feature and a cluster center of the k-th classification cluster as a reference distance; 5. The method of claim 4, further comprising: sequentially obtaining h clustering sub-image features in the at least one clustering sub-image feature based on the reference distance, and setting the h clustering sub-image features as sampling sub-image features included in the kth classification cluster, where h is a positive integer and h is less than or equal to a quantity of the at least one clustering sub-image feature.

6. The second attention sub-network includes a query weight matrix and a key weight matrix, and the step of determining a block-sparse global self-attention weight matrix of the N sampled sub-image features based on the block-sparse matrix by the global self-attention network layer includes: constructing a sampled sub-image feature matrix based on the sampled sub-image features included in each of the at least two classification clusters by the global self-attention network layer; multiplying the sampled sub-image feature matrix with the query weight matrix to obtain a query matrix; and multiplying the sampled sub-image feature matrix with the key weight matrix to obtain a key matrix; determining a block sparse global relevance matrix based on the query matrix, a transpose of the key matrix, and the block sparse matrix; and performing a normalization process on the block sparse global relevance matrix to obtain the block sparse global self-attention weight matrix.

7. The second attention sub-network further includes a value weight matrix, and the N sampled sub-image features are sampled sub-image features N ｊ wherein j is a positive integer equal to or less than N, and the step of determining, by the self-attention network layer, block sparse self-attention corresponding to each of the sampled sub-image features based on the at least two classification clusters and the block sparse global self-attention weight matrix includes: multiplying the sampled sub-image feature matrix and the value weight matrix by the self-attention network layer to obtain a value matrix; The sampled sub-image feature N ｊ a sampling sub-image feature in a classification cluster to which the target sub-image feature belongs; From the block sparse global self-attention weight matrix, ｊ and the target sampling sub-image feature as a target block sparse global self-attention weight; obtaining a value vector from the value matrix corresponding to the target sampling sub-image feature as a target value vector; Based on the target value vector and the target block sparse global self-attention weights, the sampled sub-image feature N ｊ and determining a block sparse self-attention corresponding to

8. The step of determining a classification result of the image of the detection target based on the first feature vector and the second feature vector includes: inputting the first feature vector and the second feature vector into a classification sub-network of an image discrimination model, the classification sub-network including a feature fusion network layer and a classification network layer; performing a feature fusion process on the first feature vector and the second feature vector by the feature fusion network layer to obtain a fused feature vector; The method of claim 1 , further comprising: performing a classification process on the fused feature vector by the classification network layer to obtain a classification result of the image to be detected.

9. 1. A method of image detection implemented by a computing device, comprising: obtaining a sample image and performing a feature extraction process on the sample image to obtain a sample feature representation subset of the sample image, the sample image including at least two sample sub-images, the sample feature representation subset including at least two sample sub-image features, the at least two sample sub-image features having a one-to-one correspondence with the at least two sample sub-images; inputting the at least two sample sub-images into an initial image discrimination model, generating sample attention weights corresponding to each of the at least two sample sub-image features through the initial image discrimination model, and performing a weighted aggregation process on the at least two sample sub-image features according to the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector; performing a clustering sampling process on the at least two sample sub-image features using the initial image classification model to obtain at least two sample classification clusters, the sample classification clusters including sample sampling sub-image features; determining sample block sparse self-attentions corresponding to each of the sample sampling sub-image features based on the at least two sample classification clusters and a block sparse matrix; and determining a second sample feature vector based on the at least two sample block sparse self-attentions, wherein the sample block sparse self-attentions corresponding to the sample sampling sub-image features are determined based on the sample sampling sub-image features in the sample classification cluster to which the sample sampling sub-image features belong; determining a sample classification result of the sample image based on the first sample feature vector and the second sample feature vector according to the initial image classification model; and adjusting model parameters of the initial image identification model based on the at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and classification labels corresponding to the sample images, to obtain an image identification model used to identify the classification result of the image to be detected.

10. The step of adjusting model parameters of the initial image discrimination model based on the at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and classification labels corresponding to the sample images to obtain an image discrimination model used to discriminate the classification results of the image to be detected includes: determining a divergence loss value based on the at least two sample classification clusters and sample attention weights corresponding to each of the at least two sample sub-image features; determining a classification loss value based on the sample classification results and classification labels corresponding to the sample images; performing a weighted sum of the divergence loss value and the classification loss value to obtain a model total loss value; and performing model parameter adjustment on the initial image identification model based on the model total loss value to obtain the image identification model.

11. The step of determining a divergence loss value based on the at least two sample classification clusters and sample attention weights corresponding to each of the at least two sample sub-image features comprises: obtaining an i-th sample classification cluster in the at least two sample classification clusters, where i is a positive integer and is less than or equal to a quantity of the at least two sample classification clusters; determining the sample sub-image features included in the i-th sample classification cluster as target sample sub-image features; determining a category divergence loss value corresponding to the i-th sample classification cluster based on sample attention weights corresponding to the target sample sub-image features and the quantity of the target sample sub-image features; and adding the category divergence loss values ​​corresponding to each sample classification cluster to obtain the divergence loss value.

12. determining a category divergence loss value corresponding to the i-th sample classification cluster based on sample attention weights corresponding to the target sample sub-image features and the quantity of the target sample sub-image features, obtaining a fitted attention distribution composed of sample attention weights corresponding to the target sample sub-image features; performing a normalization process on the fitting attention weight distribution to obtain a normalized fitting attention distribution; a uniform attention distribution corresponding to the quantity of the target sample sub-image features as an attention distribution label; and determining a category divergence loss value corresponding to the i-th sample classification cluster based on the normalized fitted attention distribution and the attention distribution label.

13. 1. An image detection device, comprising: a feature extraction module for acquiring an image of a target object, performing a feature extraction process on the image of the target object, and obtaining a feature representation subset of the image of the target object, the image of the target object including at least two sub-images, the feature representation subset including at least two sub-image features, the at least two sub-image features having a one-to-one correspondence with the at least two sub-images; a first vector generation module used for generating attention weights corresponding to the at least two sub-image features, and performing a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; a second vector generation module for performing a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, the classification clusters including the sampled sub-image features; determining block sparse self-attention corresponding to each of the sampled sub-image features based on the at least two classification clusters and a block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions, where the block sparse self-attention corresponding to the sampled sub-image feature is determined based on the sampled sub-image feature in the classification cluster to which it belongs; a classification module adapted to determine a classification result of the image of the detection target based on the first feature vector and the second feature vector.

14. 1. An image detection device, comprising: a sample feature extraction module used to obtain a sample image, perform a feature extraction process on the sample image, and obtain a sample feature representation subset of the sample image, the sample image including at least two sample sub-images, the sample feature representation subset including at least two sample sub-image features, the at least two sample sub-image features having a one-to-one correspondence with the at least two sample sub-images; a first sample vector generation module, which is used to input the at least two sample sub-images into an initial image discrimination model, generate sample attention weights corresponding to each of the at least two sample sub-image features through the initial image discrimination model, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector; a second sample vector generation module for performing a clustering sampling process on the at least two sample sub-image features by the initial image discrimination model to obtain at least two sample classification clusters, the sample classification clusters including sample sampling sub-image features; determining sample block sparse self-attentions corresponding to each of the sample sampling sub-image features based on the at least two sample classification clusters and a block sparse matrix; and determining a second sample feature vector based on the at least two sample block sparse self-attentions, the sample block sparse self-attentions corresponding to the sample sampling sub-image features being determined based on the sample sampling sub-image features in the sample classification cluster to which the sample sampling sub-image features belong; a sample classification module used for determining a sample classification result of the sample image according to the first sample feature vector and the second sample feature vector through the initial image identification model; and a training module used to adjust model parameters for the initial image identification model based on the at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and classification labels corresponding to the sample images, to obtain an image identification model used to identify the classification result of an image to be detected.

15. A computer device including a processor, a memory, and a network interface, A computing device, the processor being connected to the memory and the network interface, the network interface being used for providing data communication functions, the memory being used for storing program code, and the processor being used for calling the program code and thereby performing the method of any one of claims 1 to 12.

16. A computer readable storage medium having a computer program stored therein, the computer program being suitable for being loaded by a processor and for performing the method according to any one of claims 1 to 12.

17. A computer program product comprising computer programs / instructions which, when executed by a processor, are capable of performing the method according to any one of claims 1 to 12.

Images