Target image detection method and image detection model training method

Abstract

The application relates to a target image detection method and an image detection model training method, and particularly relates to the technical field of computer vision. The method comprises the following steps: acquiring a target image and a target support image; performing feature extraction on the target image and the target support image respectively, and correspondingly obtaining target features and target support features; generating attention features based on the similarity between the target features and the target support features; performing target detection processing on the target features to obtain candidate query features; performing target detection processing on the attention features to obtain candidate support features; and determining a target object in the target image based on the matching relationship between the candidate query features and the candidate support features. The above scheme makes the computer device more likely to notice the features corresponding to the target object in the processing process under the condition that the sample is small, thereby improving the accuracy of target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a target image detection method and an image detection model training method. BACKGROUND

[0002] The task of object detection is to find all objects of interest in an image, determine their categories and positions, and is one of the core problems in the field of computer vision.

[0003] At present, the target detection algorithm has been relatively mature, and good detection effects can be achieved in some real tasks. In the prior art, in order to ensure the accuracy of the target detection algorithm, a large amount of labeled data is required to train the designed deep learning model for a long time, so as to obtain model parameters with good recognition effect in the deep learning model, so that the deep learning model has good region detection and classification ability.

[0004] In the above scheme, in the actual application scene, there may be problems such as difficulty in on-site data collection, long time consumption in collection, insufficient number of pictures, and inconvenience in providing a large number of pictures to the outside, which leads to lack of training data for the algorithm, and it is difficult to ensure the detection effect of the model under the condition of small sample quantity. SUMMARY

[0005] The present application provides a target image detection method and an image detection model training method, which improves the accuracy of target detection. The technical solution is as follows.

[0006] On the one hand, a target image detection method is provided, which comprises:

[0007] Obtaining a target image and a target support image; the target support image contains a target object;

[0008] Respectively extracting features of the target image and the target support image, and correspondingly obtaining target features and target support features;

[0009] Generating attention features based on the similarity between the target features and the target support features;

[0010] Performing target detection processing on the target features to obtain candidate query features;

[0011] Performing target detection processing on the attention features to obtain candidate support features;

[0012] Determining the target object in the target image based on the matching relationship between the candidate query features and the candidate support features.

[0013] In yet another aspect, a target image detection method is provided, the method comprising:

[0014] obtaining a sample image and a sample support image; the sample support image containing a target object; the sample image containing a target annotation region;

[0015] extracting features from the sample image and the sample support image respectively by a feature extraction module in an image detection model, to obtain sample features and sample support features correspondingly;

[0016] generating a sample attention feature based on a similarity between the sample features and the sample support features;

[0017] performing target detection processing on the sample features by a target detection module in the image detection model, to obtain sample candidate query features;

[0018] performing target detection processing on the sample attention feature by the target detection module, to obtain sample candidate support features;

[0019] obtaining a matching relationship between the candidate query features and the candidate support features by a multi-head selection module in the image detection model, and outputting a predicted object region in the sample image based on the matching relationship;

[0020] training the image detection model according to the predicted object region and the target annotation region in the sample image, to detect a target image by the trained image detection model.

[0021] In yet another aspect, a target image detection device is provided, the device comprising:

[0022] a target image obtaining module, configured to obtain a target image and a target support image; the target support image containing a target object;

[0023] a feature extraction module, configured to extract features from the target image and the target support image respectively, to obtain target features and target support features correspondingly;

[0024] an attention feature generation module, configured to generate an attention feature based on a similarity between the target features and the target support features;

[0025] a first detection module, configured to perform target detection processing on the target features, to obtain candidate query features;

[0026] a second detection module, configured to perform target detection processing on the attention feature, to obtain candidate support features;

[0027] The target object determination module is configured to determine the target object in the target image based on a matching relationship between the candidate query feature and the candidate support feature.

[0028] In a possible implementation, the feature extraction module is further configured to:

[0029] The feature extraction module in the image detection model is configured to perform feature extraction on the target image to obtain the target feature.

[0030] The feature extraction module is configured to perform feature extraction on at least two target support images to obtain at least two intermediate features.

[0031] The mean of the at least two intermediate features is obtained as the target support feature.

[0032] In a possible implementation, the attention feature generation module is further configured to:

[0033] The target feature and the target support feature are convoluted in each dimension, and the convolution results in each dimension are constructed as the attention feature.

[0034] In a possible implementation, the target object determination module is further configured to:

[0035] The similarity between the candidate query feature and the candidate support feature is obtained as a confidence of the candidate support feature.

[0036] When the confidence of the candidate support feature is greater than a confidence threshold, the candidate support feature is determined as a region of the target object, and the confidence is obtained as a presence probability of the target object.

[0037] In still another aspect, an image detection model training apparatus is provided, and the apparatus includes:

[0038] A sample image acquisition module is configured to acquire a sample image and a sample support image; the sample support image contains a target object; and the sample image contains a target annotation region.

[0039] A sample extraction module is configured to perform feature extraction on the sample image and the sample support image, respectively, by a feature extraction module in an image detection model, to correspondingly obtain a sample feature and a sample support feature.

[0040] A sample attention feature generation module is configured to generate a sample attention feature based on a similarity between the sample feature and the sample support feature.

[0041] The first sample detection module is configured to perform target detection on the sample feature by a target detection module in the image detection model to obtain a sample candidate query feature.

[0042] The second sample detection module is configured to perform target detection on the sample attention feature by the target detection module to obtain a sample candidate support feature.

[0043] The region prediction module is configured to acquire a matching relationship between the candidate query feature and the candidate support feature by a multi-head selection module in the image detection model, and output a predicted object region in the sample image based on the matching relationship.

[0044] The model training module is configured to train the image detection model according to the predicted object region and a target annotation region in the sample image, so as to detect a target image by the trained image detection model.

[0045] In a possible implementation, the model training module is further configured to,

[0046] obtain a first loss function value according to the predicted object region and the target annotation region, and perform back propagation update on the target detection module;

[0047] obtain a second loss function value according to the predicted object region and the target annotation region, and perform back propagation update on the multi-head selection module.

[0048] In another aspect, a computer device is provided, which comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the above-mentioned target image detection method.

[0049] Alternatively, the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the above-mentioned image detection model training method.

[0050] In yet another aspect, a computer readable storage medium is provided, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned target image detection method.

[0051] Alternatively, the at least one instruction is loaded and executed by the processor to implement the above-mentioned image detection model training method.

[0052] In still another aspect, a computer program product is provided, and a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the target image detection method or the image detection model training method described above.

[0053] The technical solutions provided in the present application can include the following beneficial effects:

[0054] When detecting the target image, since the sample data on site is less, the computer device can obtain the target support image, and then perform feature extraction on the target image and the target support image, and generate attention features according to the similarity between the target features and the target support features. The computer device performs target detection on the attention features, and when the target detection model performs target detection on the attention features, it not only considers the target object in the target support image, but also considers the similar part in the target image. Therefore, the candidate support features are more likely to notice the similar part in the target image. The candidate query features are obtained by target detection according to the target features, and are the features of the position of the target object in the target image. The computer device detects the target object in the target image according to the matching relationship between the candidate query features and the candidate support features, and also judges the similarity of the target object in the target support image. Even in the case of small sample, the computer device is more likely to notice the features corresponding to the target object in the processing process, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the description of the specific embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0056] Figure 1 It is a structural schematic diagram of a target image detection system according to an exemplary embodiment.

[0057] Figure 2 It is a method flowchart of a target image detection method according to an exemplary embodiment.

[0058] Figure 3 It is a method flowchart of a target image detection method according to an exemplary embodiment.

[0059] Figure 4 is a method flow chart of an image detection model training method according to an example embodiment.

[0060] Figure 5 A structural schematic diagram of an image detection model related to an embodiment of the present application is shown.

[0061] Figure 6 is a flow chart of a target image detection method according to an example embodiment.

[0062] Figure 7 is a structural block diagram of a target image detection device according to an example embodiment.

[0063] Figure 8 is a structural block diagram of an image detection model training device according to an example embodiment.

[0064] Figure 9 is a schematic diagram of a computer device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions of the present application will be described clearly and completely below in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0066] It should be understood that the "indication" mentioned in the embodiments of the present application can be direct indication, or indirect indication, or can be an indication of an associated relationship. For example, A indicates B, which can mean that B can be obtained through A; or it can mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; or it can mean that A and B have an associated relationship.

[0067] In the description of the embodiments of the present application, the term "corresponding" can mean a direct or indirect corresponding relationship between the two, or an associated relationship between the two, or an indication and being indicated, configuration and being configured, etc.

[0068] In the embodiments of the present application, "predefined" can be realized by pre-saving corresponding codes, tables or other means for indicating related information in devices (such as terminal devices and network devices), and the present application does not limit the specific implementation manner.

[0069] Before the various embodiments shown in the present application are described, several concepts related to the present application are first introduced.

[0070] 1) AI (Artificial Intelligence)

[0071] Artificial Intelligence, English abbreviation AI. It is a new technology science of studying, developing theory, method, technology and application system for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Since the birth of artificial intelligence, the theory and technology have become more mature, and the application field has also been expanding. It can be imagined that the technology products brought by artificial intelligence in the future will be the "container" of human intelligence. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is not human intelligence, but can think like a person and may exceed human intelligence.

[0072] The main material basis for studying artificial intelligence and the machine that can realize the artificial intelligence technology platform is the computer. In addition to computer science, artificial intelligence also involves information theory, control theory, automation, bionics, biology, psychology, mathematical logic, linguistics, medicine and philosophy, etc. The main content of artificial intelligence research includes: knowledge representation, automatic reasoning and search method, machine learning and knowledge acquisition, knowledge processing system, natural language understanding, computer vision, intelligent robot, automatic program design, etc.

[0073] 2) CV (Computer Vision)

[0074] Computer vision is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics to make computer processing more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish an artificial intelligence system that can obtain 'information' from images or multidimensional data. The information referred to here refers to the information defined by Shannon, which can be used to help make a "decision". Because perception can be regarded as extracting information from sensory signals, computer vision can also be regarded as a science that studies how artificial systems "perceive" from images or multidimensional data.

[0075] 3) Machine Learning (ML)

[0076] Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure, and continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. It is applied in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0077] The target image detection method provided in the embodiments of the present application can be applied to a computer device with strong data processing capability. The target image detection method can be a method for identifying and detecting target objects by using an image detection model after training the image detection model. The image detection model can process input features to obtain the region where the target object is located and the category of the target object corresponding to the input features. In a possible implementation manner, the target image detection method provided in the embodiments of the present application can be applied to a personal computer, a workstation, or a server, and the training of the image detection model can be performed by at least one of the personal computer, the workstation, and the server.

[0078] Figure 1 FIG. 1 is a structural schematic diagram of a target image detection system according to an example embodiment. The target image detection system includes a server 110 and a terminal 120. The terminal 120 and the server 110 perform data communication through a communication network. The communication network can be a wired network or a wireless network.

[0079] Optionally, the terminal 120 is installed with an application program having an image processing function. The application program can be a professional image processing application program, a social application program, a virtual reality application program, or an AI application program having an image processing function. The embodiments of the present application do not limit this.

[0080] Optionally, the terminal 120 can be a terminal device having an image acquisition component. The image acquisition component is used to acquire images and store the images in a data storage module in the terminal 120. The terminal 120 can also be a terminal device having a data transmission interface. The data transmission interface is used to receive image data collected by an image acquisition device having an image acquisition component.

[0081] Optionally, the terminal 120 can be a mobile terminal such as a smartphone, a tablet computer, a laptop computer, etc. The terminal 120 can also be a desktop computer, a projection computer, or a smart terminal having a data processing component. The embodiments of the present application do not limit this.

[0082] The server 110 can be implemented as a single server or a server cluster composed of a plurality of servers, which can be physical servers or cloud servers. In a possible implementation, the server 110 is a background server of an application program in the terminal 120.

[0083] In a possible implementation of the embodiments of the present application, the server 110 trains the image detection model by using a pre-set training sample set (including various training sample images), which can include sample images of different categories, and each training sample image has its own category label information (i.e., label value). When the server 110 completes the training process of the image detection model, the trained image detection model is sent to the terminal 120 through a wired network or a wireless network.

[0084] The terminal 120 receives the trained classification network model and sends the data information (such as weight information) corresponding to the image detection model to the application program with the identification and classification function, so that when the user uses the application program, the input image can be identified and classified.

[0085] Optionally, the server can be a physical server, a server cluster composed of a plurality of physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms.

[0086] Optionally, the system can further include a management device for managing the system (such as managing the connection state between each module and the server), and the management device and the server are connected through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0087] Optionally, the wireless network or wired network uses standard communication techniques and / or protocols. The network is usually the Internet, but can also be any other network, including but not limited to a local area network, a metropolitan area network, a wide area network, a mobile, limited or wireless network, a private network or any combination of virtual private networks. In some embodiments, technologies and / or formats such as hypertext markup language, extensible markup language, etc. are used to represent data exchanged through the network. In addition, all or some links can be encrypted using conventional encryption technologies such as secure sockets layer, transport layer security, virtual private network, Internet protocol security, etc. In other embodiments, custom and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies.

[0088] Figure 2is a method flowchart of a target image detection method according to an exemplary embodiment. The method is executed by a computer device, which can be a server or a terminal in a target image detection system as shown in Figure 1 Figure 2 The target image detection method can include the following steps:

[0089] Step 201, obtaining a target image and a target support image.

[0090] In the embodiments of the present application, the target support image contains the target object, and the scheme of the embodiments of the present application ultimately needs to achieve detection of the presence or absence and the position of the target object in the target image.

[0091] Therefore, in the embodiments of the present application, the target image can be considered as an image to be detected, and the target support image is a pre-prepared image that plays an auxiliary role in the detection process of the target image.

[0092] Step 202, respectively extracting features from the target image and the target support image, and obtaining target features and target support features.

[0093] After the computer device obtains the target image and the target support image, the computer device can perform feature extraction processing on the target image to obtain target features corresponding to the target image, and can also perform feature extraction processing on the target support image to obtain target support features.

[0094] In a possible implementation, the computer device can use a pre-trained feature extraction network, such as ResNet-50, to respectively extract features from the target image and the target support image.

[0095] Since the ultimate detection target in the embodiments of the present application is the presence or absence and possible position of the target object in the target image, the extracted features should also focus on the target object. Therefore, in a possible implementation, the feature extraction network can be trained by taking the image as a sample and taking the position of the target object on the image as a label, at this time the trained feature extraction network is more likely to pay attention to the features of the target object and is more inclined to extract the features of the target object in the feature extraction process.

[0096] Step 203, generating an attention feature based on the similarity between the target features and the target support features.

[0097] After obtaining the target features and the target support features, the computer device can obtain the similarity between the target features and the target support features, and generate an attention feature based on the similarity.​

[0098] Optionally, the similarity between the target feature and the target support feature can be obtained by a deep convolution method. That is, the target feature and the target support feature are respectively convolved according to each channel to obtain a feature map of each channel to constitute the attention feature.

[0099] At this time, the values in the attention feature are different from the original target support feature, and not only focus on part of the target object, but also focus more on the similar part between the target feature and the target support feature, that is, the similar part between the target feature and the target object.

[0100] Step 204, performing target detection processing on the target feature to obtain a candidate query feature.

[0101] Optionally, the computer device can perform target detection processing on the target feature by using the trained target detection model, generate a candidate box in the target feature, and obtain the feature in the candidate box as the candidate query feature. At this time, the candidate query feature indicates the region where the target object may exist as judged by the target feature.

[0102] Step 205, performing target detection processing on the attention feature to obtain a candidate support feature.

[0103] Optionally, the computer device also performs target detection processing on the attention feature by using the trained target detection model, thereby generating a candidate box in the attention feature, and obtaining the feature in the candidate box as the candidate support feature. At this time, the candidate support feature indicates the region where the target object may exist as judged by the attention feature.

[0104] Step 206, determining the target object in the target image based on the matching relationship between the candidate query feature and the candidate support feature.

[0105] In one possible implementation, after obtaining the candidate query feature and the candidate support feature, the confidence of each candidate query feature can be determined according to the matching relationship between the candidate support feature and the candidate support feature, and the region corresponding to the candidate query feature with a confidence higher than a threshold value is determined as the region corresponding to the target object, so as to determine the target object in the target image.

[0106] Since the candidate support feature is the feature in the detection box obtained by relying on the similarity between the target support image and the target image, it can be logically considered that the candidate support feature represents the feature of the similar region in the target image found by the target support feature (i.e., the feature of the target object) extracted from the target support image.

[0107] The candidate query feature is only dependent on the feature in the detection box obtained from the target image. When the matching degree of the candidate query feature and the candidate support feature is high, it can be considered that the feature in the candidate query feature has high similarity with the target object, and thus the region where the candidate query feature is located can be regarded as the region where the target object is located.

[0108] In summary, when detecting the target image, the computer device can obtain the target support image due to the small amount of sample data in the field. The target image and the target support image are subjected to feature extraction, and the similarity between the target feature and the target support feature is generated to obtain the attention feature. The computer device performs target detection on the attention feature. When the target detection model performs target detection on the attention feature, the target object in the target support image is considered, and the similar part in the target image is also considered. Therefore, the candidate support feature is more likely to pay attention to the similar part in the target image. The candidate query feature is obtained by target detection based on the target feature, and is the feature of the position where the target object may be located in the target image. The computer device detects the target object in the target image by matching the candidate query feature and the candidate support feature, and also assists in judging the similarity of the target object in the target support image. Even in the case of small sample, the computer device is more likely to pay attention to the feature corresponding to the target object in the processing process, thereby improving the accuracy of target detection.

[0109] Figure 3 FIG. 1 is a flowchart of a target image detection method according to an example embodiment. The method is performed by a computer device, which can be a server or a terminal in a target image detection system as shown in FIG. 2. Figure 1 As shown in FIG. 3, the target image detection method can include the following steps. Figure 3

[0110] Step 301: Obtain a target image and a target support image.

[0111] In a possible implementation manner of the embodiment, the target image is an image collected in an engineering scene and needs to be detected. The computer device needs to detect the target image to determine whether the target object exists in the target image.

[0112] The target support image is an image collected in an engineering scene similar to the target image, and the target object is included in the target support image.

[0113] Optionally, the target support image can be a plurality of target support images. The features in the plurality of target support images can assist in the detection of the target image in the subsequent process, and the detection accuracy of the target image can be further improved.​

[0114] At step 302, the feature extraction module in the image detection model is used to extract features of the target image, and the target features are obtained.

[0115] In a possible implementation of the embodiments of the present application, the feature extraction module can be pre-trained.

[0116] Further, the feature extraction module can be trained by sample images collected in an engineering scene similar to the target support image.

[0117] For example, when the feature extraction module needs to be trained, a machine learning model containing the feature extraction module can be constructed (such as a target detection model based on the feature extraction module), the computer device trains the target detection model by using sample images and the bounding box of the target object on the sample images, and the parameters of the feature extraction module of the trained target detection model are extracted as the parameters of the feature extraction module in the target detection model.

[0118] The feature extraction model trained in the above manner can pay more attention to the features of the target object, and the extracted features have a higher correlation with the target object.

[0119] At step 303, the feature extraction module is used to extract features of at least two target support images, and at least two intermediate features are obtained.

[0120] In the embodiments of the present application, when there are multiple target support images, the computer device can extract features of each target support image by using the feature extraction module, and thus obtain each intermediate feature. At this time, the intermediate feature represents the features of the target object in each target support image.

[0121] At step 304, the mean of the at least two intermediate features is obtained as the target support feature.

[0122] When the intermediate features corresponding to each target support image are obtained, since there are still differences between similar target objects in actual application scenarios (i.e., engineering scenarios), in order to improve the accuracy of detecting target objects in different scenarios, the intermediate features extracted from different target support images can be fused when the target support feature is obtained, that is, the mean of the intermediate features is obtained as the target support feature, so that the obtained target support feature is obtained by considering the image conditions of the target objects in different scenarios.

[0123] At step 305, the target feature and the target support feature are convolved in each dimension, and the convolution results in each dimension are constructed as the attention feature.

[0124] After the target feature and the target support feature are extracted by the feature extraction module, the computer device can perform convolution processing on the target feature and the target support feature in each dimension (i.e., each channel, such as RGB three channels) respectively, so as to calculate the similarity between the target feature and the target support feature, and construct the convolution result as an attention feature.

[0125] For example, the generation principle of the attention feature is as follows:

[0126]

[0127] wherein X e t S×S×C represents the support feature; X e t H×W×C represents the query feature; therefore, according to the above convolution formula, a feature matrix G h,w,c , i.e., the attention feature, can be generated.

[0128] In step 306, the target feature is subjected to target detection processing to obtain a candidate query feature.

[0129] After the target feature is extracted from the target image by the feature extraction module, the computer device can perform target detection on the target feature by using a target detection algorithm, i.e., performing data processing on the target feature by using a trained region generation network, so as to output a candidate box (i.e., a region of interest) corresponding to the target feature, and the feature in the candidate box is the candidate query feature.

[0130] In one possible implementation, the region generation network is trained by using a sample image as a sample and a label box of a target object on the sample image as a label.

[0131] Therefore, the region generation network can also pay more attention to the region corresponding to the target object and generate a corresponding candidate box, and the feature in the candidate box is the feature of the target object predicted from the target feature, which is used as the candidate query feature.

[0132] In step 307, the attention feature is subjected to target detection processing to obtain a candidate support feature.

[0133] In one possible implementation, the computer device can perform data processing on the attention feature by using a trained region generation network, so as to output a candidate box corresponding to the target feature, and the feature in the candidate box is the candidate support feature.

[0134] At this time, the candidate support feature is determined according to the attention feature, and the data of the attention feature tends to represent the similarity between each region of the target image and the target support image.

[0135] Therefore, through the region generation network, the candidate box obtained by data processing the attention feature tends to select a similar region of the target object in the target image and the target support image.

[0136] In step 308, the target object is determined in the target image based on the matching relationship between the candidate query feature and the candidate support feature.

[0137] Based on the above steps, when the matching relationship between the candidate query feature and the candidate support feature meets the condition (for example, when the candidate query feature matches the candidate support feature), it means that the candidate query feature is consistent with the candidate support feature obtained by the similarity between the target support image and the target image, and the candidate query feature has a greater possibility of being the feature corresponding to the target object.

[0138] Therefore, the computer device can determine the region corresponding to the candidate query feature in the target image as the region where the target object is located, thereby determining the target object in the target image.

[0139] In a possible implementation, the similarity between the candidate query feature and the candidate support feature is obtained as the confidence of the candidate support feature; when the confidence of the candidate support feature is greater than a confidence threshold, the candidate support feature is determined as the region of the target object, and the confidence is obtained as the existence probability of the target object.

[0140] When the similarity between the candidate query feature and the candidate support feature is greater, it means that the candidate query feature is more likely to be the feature corresponding to the target object, and therefore the similarity between the candidate query feature and the candidate support feature is obtained as the confidence of the candidate support feature. When the region of the candidate support feature with high confidence is more likely to exist the target object, the region is determined as the region of the target object.

[0141] In a possible implementation, the similarity between the candidate query feature and the candidate support feature is obtained according to at least one of global correlation, local correlation, and image block matching relationship between the candidate query feature and the candidate support feature.

[0142] The global correlation between the candidate query feature and the candidate support feature is obtained by stacking the candidate query feature and the candidate support feature after vector transformation of the dimensions together and inputting into a fully connected layer.

[0143] The local correlation between the candidate query feature and the candidate support feature is that the candidate query feature and the candidate support feature are respectively sent into a convolution layer to further extract features, then the obtained query feature is slid on the obtained support feature, the cross-correlation feature is calculated and nonlinearized, and the local similarity is calculated.

[0144] The image block matching relationship between the candidate query feature and the candidate support feature is that the candidate query feature and the candidate support feature are stacked together after vector transformation of the dimensions, input into a convolution layer, a nonlinear layer and a pooling layer, and the image block similarity is calculated.

[0145] In a possible implementation, the global similarity, the local similarity and the image block matching relationship between the candidate query feature and the candidate support feature are respectively acquired and weightedly summed, so as to obtain the similarity between the candidate query feature and the candidate support feature.

[0146] In summary, when the target image is detected, the computer device can acquire the target support image due to less sample data on site, and then the target image and the target support image are subjected to feature extraction, and the similarity between the target feature and the target support feature is generated to obtain the attention feature. The computer device performs target detection processing on the attention feature, and when the target detection model performs target detection on the attention feature, the target object in the target support image is considered, and the similar part in the target image is also considered. Therefore, the candidate support feature is more likely to notice the similar part in the target image. The candidate query feature is obtained by target detection based on the target feature, and is the feature of the position of the target object in the target image. The computer device detects the target object in the target image according to the matching relationship between the candidate query feature and the candidate support feature, and also assists in judging the similarity of the target object in the target support image. Even in the case of small sample, the computer device is more likely to notice the feature corresponding to the target object in the processing process, thereby improving the accuracy of target detection.

[0147] Figure 4 FIG. 1 is a method flowchart of an image detection model training method according to an example embodiment. The method is performed by a computer device, which can be a server or a terminal in a target image detection system as shown in FIG. 2. Figure 1 As shown in FIG. 3, the image detection model training method can include the following steps: Figure 4

[0148] Step 401, acquiring a sample image and a sample support image.

[0149] The sample support image contains a target object. The sample image contains a target annotation region;​

[0150] At step 402, the feature extraction module in the image detection model is used to extract features from the sample image and the sample support image respectively, so as to obtain sample features and sample support features.

[0151] At step 403, the sample attention features are generated based on the similarity between the sample features and the sample support features.

[0152] At step 404, the target detection module in the image detection model is used to perform target detection on the sample features, so as to obtain sample candidate query features.

[0153] At step 405, the target detection module is used to perform target detection on the sample attention features, so as to obtain sample candidate support features.

[0154] At step 406, the multi-head selection module in the image detection model is used to obtain the matching relationship between the candidate query features and the candidate support features, and output the predicted object region in the sample image based on the matching relationship.

[0155] In the model application process, steps 401 and 406 are the model training process. Figure 3 The model application process in the embodiment shown in the figure is relative to the model training process. Since the data processing manner of the model application process is similar to that of the model training process, details are not described here.

[0156] At step 407, the image detection model is trained according to the predicted object region and the target annotation region in the sample image, so as to detect target images by using the trained image detection model.

[0157] In one possible implementation, a first loss function value is obtained according to the predicted object region and the target annotation region, so as to update the target detection module by back propagation.

[0158] A second loss function value is obtained according to the predicted object region and the target annotation region, so as to update the multi-head selection module by back propagation.

[0159] Please refer to Figure 5 which shows a structure diagram of an image detection model according to an embodiment of the present application. As shown in the figure, Figure 5As shown, the network infrastructure used by the image detection model is ResNet-50. The RPN part uses the standard RPN used in Faster R-CNN, specifically including 1 convolutional layer of 3x3 and 2 convolutional layers of 1x1. The Head part calculates the features of each candidate region. For the support image, the feature region is directly selected by the GT (ground truth, i.e., the labeled box) detection box, and after ROI-Align, the Res5 block is used to extract the features. For the query image, the predicted detection box position is determined by the RPN network, and after ROI-Align in the predicted detection box position, the Res5 block is used to extract the features. When predicting the detection box position and the target category, multiple convolutional layers and fully connected layers are used to calculate the global relationship, local relationship and image block matching relationship of the support features and query features, and the final confidence is the sum of the three.

[0160] During training of the above model, the positioning loss in the RPN stage is selected as smooth l1 loss, and the classification loss is selected as binary cross-entropy loss (the first loss function value can be obtained by smooth l1 loss and binary cross-entropy loss). The classification loss of the Head part is selected as softmax cross-entropy loss, and the detection box regression loss is selected as smooth l1 loss (the first loss function value can be obtained by smooth l1 loss and softmax cross-entropy loss). The optimizer in the training process uses the SGD algorithm with momentum, sets the BatchSize to 4, the total number of iterations to 120000, the initial learning rate to 0.002, and the learning rate to 1 / 10 at the 80000th and 100000th times, respectively. The training process is trained on 2 RTX2080 graphics cards, and the network parameters are updated through the back propagation algorithm.

[0161] To sum up, when detecting the target image, the computer device can acquire the target support image due to less sample data on site, and then performs feature extraction on the target image and the target support image, and generates attention features based on the similarity between the target features and the target support features. The computer device performs target detection on the attention features. When the target detection model detects the attention features, it not only considers the target object in the target support image, but also considers the similar part in the target image. Therefore, the candidate support features are more likely to notice the similar part in the target image. The candidate query features are obtained by target detection based on the target features, and are the features of the position of the target object in the target image. The computer device detects the target object in the target image based on the matching relationship between the candidate query features and the candidate support features, and also judges the similarity of the target object in the target support image. Even in the case of small samples, the computer device is more likely to notice the features corresponding to the target object in the processing process, thereby improving the accuracy of target detection.

[0162] In addition, compared with the conventional target detection model, the model trained by the embodiment of the present application generates the attention features based on the similarity between the target features and the target support features under the premise of detecting the candidate box, generates the candidate box as a reference, and verifies the features in the candidate box of the target image based on the features in the reference candidate box, thereby further improving the accuracy of target detection.

[0163] Figure 6 FIG. 1 is a flow chart of a target image detection method according to an example embodiment. The method is performed by a computer device, which can be a server or a terminal in a target image detection system as shown in FIG. 2. In the embodiment of the present application, the target image detection method includes a training method of an image detection model. As shown in FIG. 3, the target image detection method is applied to the garbage dumping detection scene in engineering, and the target image detection method is as follows. Figure 1 Figure 6

[0164] First, in the embodiment of the present application, the computer device trains an algorithm model on the COCO open source image data set.

[0165] ​​In the actual scene test stage, first, 20 pictures of construction waste dumping are collected as support pictures, and the support features are extracted by using the trained model; then, after the video captured by the monitoring camera is frame-extracted into images, the query features are extracted by ResNet-50, the similarity of the query features and the support features is calculated, the candidate frame is input into RPN to generate a candidate frame, the detection confidence is calculated by the Head part, and finally the construction waste detection situation is output, so as to efficiently judge whether there is construction waste dumping. The code involved in the present application is written in Python3 language, and is trained using the Pytorch deep learning framework.

[0166] The processing flowchart of the embodiment of the present application is shown in Figure 6 The specific implementation steps are as follows:

[0167] 1) Data set preprocessing

[0168] The algorithm model is trained on the COCO open source image data set. In order to make the network model obtained by training have good generalization ability, data augmentation is performed on the training data. First, the image size is scaled to 512*288. Then, random translation, rotation and scale transformation are performed. Specifically:

[0169]

[0170] Wherein: (x, y, 1) represents the matrix before image translation; (x', y', 1) represents the matrix after image translation; d x and d y respectively

[0171] are the pixel amounts of image translation in the x-axis and y-axis;

[0172]

[0173] Wherein: (x, y, 1) represents the matrix before image scale transformation; (x”, y”, 1) represents the matrix after image scale transformation;

[0174] s x and s y are the scale transformation factors of the image in the x-axis and y-axis, respectively;

[0175]

[0176] Wherein: (x, y, 1) represents the matrix before image rotation; (x”', y”', 1) represents the matrix after image rotation; is the rotation angle.

[0177] 2) Building of algorithm network

[0178] The network infrastructure used by the application is ResNet-50. The RPN part adopts the standard RPN used in Faster R-CNN, specifically including 1 3x3 convolution layer and 2 1x1 convolution layers. When the Head part calculates the features of each candidate region, Res5 block is used after ROI-Align to extract the features; when the position of the detection frame and the target category are predicted, multiple convolution layers and fully connected layers are used to calculate the global relationship, local relationship and image block matching relationship of support features and query features, and the final confidence is the sum of the three.

[0179] 3) Training of algorithm network

[0180] When the algorithm model of the application is trained, the positioning loss in the RPN stage is selected as smooth l1 loss, the classification loss is selected as binary cross-entropy loss, the classification loss of the Head part is selected as softmax cross-entropy loss, and the detection frame regression loss is selected as smooth l1 loss. The optimizer in the training process uses the SGD algorithm with momentum, sets BatchSize to 4, the total number of iterations to 120000 times, the initial learning rate to 0.002, and the learning rate to 1 / 10 at the 80000th and 100000th times. The training process is trained on 2 RTX2080 graphics cards, and the network parameters are updated through the back propagation algorithm.

[0181] 4) Collection of support images

[0182] In a similar scene to the on-site use, 20 pictures of construction waste dumping are collected as support pictures.

[0183] 5) Extraction of support features

[0184] With the model obtained by algorithm model training, the support features corresponding to the 20 support pictures are extracted, and the mean value of the features of all pictures is taken as the final support feature of the construction waste category.

[0185] 6) Test image preprocessing

[0186] After extracting the video shot by the monitoring camera into an image, the image size is normalized to 640x640, which facilitates faster model inference and improves algorithm performance.

[0187] 7) Test image input model

[0188] The 640x640 size test image and the constructed support feature are input into the trained model, and the model outputs the detection result of the construction waste in the test image.

[0189] 8) judging whether there is construction waste dumping

[0190] According to the construction waste detection result and the confidence, it is judged whether there is construction waste dumping.

[0191] Figure 7 A structural block diagram of a target image detection device according to an exemplary embodiment is shown.

[0192] The device comprises:

[0193] The target image acquisition module 701 is configured to acquire a target image and a target support image, and the target support image contains a target object.

[0194] The feature extraction module 702 is configured to perform feature extraction on the target image and the target support image respectively, and obtain target features and target support features correspondingly.

[0195] The attention feature generation module 703 is configured to generate an attention feature based on the similarity between the target features and the target support features.

[0196] The first detection module 704 is configured to perform target detection processing on the target features to obtain candidate query features.

[0197] The second detection module 705 is configured to perform target detection processing on the attention features to obtain candidate support features.

[0198] The target object determination module 706 is configured to determine the target object in the target image based on the matching relationship between the candidate query features and the candidate support features.

[0199] In a possible implementation, the feature extraction module is further configured to:

[0200] The feature extraction module in the image detection model is used to perform feature extraction on the target image to obtain the target features.

[0201] The feature extraction module is used to perform feature extraction on at least two target support images to obtain at least two intermediate features.

[0202] The mean value of the at least two intermediate features is obtained as the target support features.

[0203] In a possible implementation, the attention feature generation module is further configured to:

[0204] The target features and the target support features are convoluted in each dimension, and the convolution results in each dimension are constructed as the attention features.

[0205] In a possible implementation, the target object determination module is further configured to:

[0206] obtain the similarity between the candidate query feature and the candidate support feature as a confidence of the candidate support feature;

[0207] when the confidence of the candidate support feature is greater than a confidence threshold, determine the candidate support feature as a region of the target object, and obtain the confidence as a presence probability of the target object.

[0208] In summary, when detecting a target image, the computer device can obtain a target support image, and perform feature extraction on the target image and the target support image, and generate attention features based on the similarity between the target features and the target support features. The computer device performs target detection on the attention features. When the target detection model performs target detection on the attention features, it not only considers the target object in the target support image, but also considers the similar part in the target image. Therefore, the candidate support feature is more likely to notice the similar part in the target image. The candidate query feature is obtained by target detection based on the target features, and is the feature of the position of the target object in the target image. The computer device detects the target object in the target image based on the matching relationship between the candidate query feature and the candidate support feature, and also assists in judging the similarity between the target object in the target support image. Even in the case of small samples, the computer device is more likely to notice the features corresponding to the target object in the processing process, thereby improving the accuracy of target detection.

[0209] Figure 8 FIG. 1 is a structural block diagram of an image detection model training device according to an example embodiment. The device includes:

[0210] a sample image acquisition module 801 configured to acquire a sample image and a sample support image; the sample support image contains a target object; and the sample image contains a target annotation region;

[0211] a sample extraction module 802 configured to perform feature extraction on the sample image and the sample support image respectively by using a feature extraction module in an image detection model, and obtain sample features and sample support features correspondingly;

[0212] a sample attention feature generation module 803 configured to generate sample attention features based on the similarity between the sample features and the sample support features;

[0213] The first sample detection module 804 is configured to perform target detection on the sample feature by a target detection module in the image detection model to obtain a sample candidate query feature.

[0214] The second sample detection module 805 is configured to perform target detection on the sample attention feature by the target detection module to obtain a sample candidate support feature.

[0215] The region prediction module 806 is configured to acquire a matching relationship between the candidate query feature and the candidate support feature by a multi-head selection module in the image detection model, and output a predicted object region in the sample image based on the matching relationship.

[0216] The model training module 807 is configured to train the image detection model according to the predicted object region and a target annotation region in the sample image, so as to detect a target image by the trained image detection model.

[0217] In a possible implementation, the model training module is further configured to,

[0218] According to the predicted object region and the target annotation region, a first loss function value is acquired to update the target detection module by back propagation.

[0219] According to the predicted object region and the target annotation region, a second loss function value is acquired to update the multi-head selection module by back propagation.

[0220] In summary, when detecting a target image, the computer device can acquire a target support image due to less sample data on site, and perform feature extraction on the target image and the target support image, and generate an attention feature according to the similarity between the target feature and the target support feature. The computer device performs target detection on the attention feature, and when the target detection model performs target detection on the attention feature, the target detection model considers not only the target object in the target support image but also the similar part in the target image. Therefore, the candidate support feature is more likely to notice the similar part in the target image. The candidate query feature is obtained by target detection on the target feature, and is the feature of the position of the target object in the target image. The computer device detects the target object in the target image according to the matching relationship between the candidate query feature and the candidate support feature, and also judges the similarity with the target object in the target support image. Even in the case of small sample, the computer device is more likely to notice the feature corresponding to the target object in the processing process, thereby improving the accuracy of target detection.

[0221] Please refer to Figure 9Fig. 1 is a schematic diagram of a computer device according to an example embodiment of the present application. The computer device includes a memory and a processor. The memory is configured to store a computer program. The computer program, when executed by the processor, implements the method described above.

[0222] The processor can be a Central Processing Unit (CPU). The processor can also be other general-purpose processors, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0223] The memory is a non-transitory computer readable storage medium configured to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiments of the present application. The processor executes various functions and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, i.e. implements the method in the method embodiments described above.

[0224] The memory can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function. The data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely located with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0225] In an exemplary embodiment, a computer readable storage medium storing at least one computer program is also provided, the at least one computer program is loaded and executed by a processor to implement all or part of the steps in the above method. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0226] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0227] It is to be understood that the application is not limited to the precise details of design and construction that have been described and exemplified above and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. A target image detection method, characterized by, The method comprises: obtaining a target image and a target support image; the target support image contains a target object; respectively extracting features from the target image and the target support image to obtain target features and target support features; generating attention features based on the similarity between the target features and the target support features; performing target detection processing on the target features to generate a candidate box in the target features, and obtaining features in the candidate box as candidate query features; performing target detection processing on the attention features to generate a candidate box in the attention features, and obtaining features in the candidate box as candidate support features; determining the target object in the target image based on the matching relationship between the candidate query features and the candidate support features.

2. The method of claim 1, wherein, The feature extraction from the target image and the target support image to obtain the target features and the target support features comprises: extracting features from the target image by a feature extraction module in an image detection model to obtain the target features; extracting features from at least two target support images by the feature extraction module to obtain at least two intermediate features; obtaining the mean of the at least two intermediate features as the target support features.

3. The method of claim 1, wherein, The generation of the attention features based on the similarity between the target features and the target support features comprises: performing convolution processing on the target features and the target support features in each dimension, and constructing the convolution results in each dimension as the attention features.

4. The method of claim 1, wherein, The determination of the target object in the target image based on the matching relationship between the candidate query features and the candidate support features comprises: obtaining the similarity between the candidate query features and the candidate support features as the confidence of the candidate support features; when the confidence of the candidate support features is greater than a confidence threshold, determining the candidate support features as the region of the target object, and obtaining the confidence as the existence probability of the target object.

5. An image detection model training method, characterized in that, The method comprises: obtaining a sample image and a sample support image; the sample support image contains a target object; the sample image contains a target annotation region; respectively extracting features from the sample image and the sample support image by a feature extraction module in an image detection model to obtain sample features and sample support features; generating sample attention features based on the similarity between the sample features and the sample support features; performing target detection processing on the sample features by a target detection module in the image detection model to generate a candidate box in the sample features, and obtaining features in the candidate box as sample candidate query features; performing target detection processing on the sample attention features by the target detection module to generate a candidate box in the sample attention features, and obtaining features in the candidate box as sample candidate support features; obtaining the matching relationship between the candidate query features and the candidate support features by a multi-head selection module in the image detection model, and outputting a predicted object region in the sample image based on the matching relationship. According to the predicted object region and the target annotation region in the sample image, the image detection model is trained to detect a target image through the trained image detection model.

6. The method of claim 5, wherein, The training of the image detection model according to the predicted object region and the target annotation region in the sample image comprises: According to the predicted object region and the target annotation region, a first loss function value is obtained to update the target detection module through back propagation; According to the predicted object region and the target annotation region, a second loss function value is obtained to update the multi-head selection module through back propagation.

7. An object image detection apparatus characterized by comprising: The device comprises: A target image acquisition module is configured to acquire a target image and a target support image, wherein the target support image contains a target object. A feature extraction module is configured to perform feature extraction on the target image and the target support image respectively, and obtain target features and target support features correspondingly. An attention feature generation module is configured to generate an attention feature based on the similarity between the target features and the target support features. A first detection module is configured to perform target detection processing on the target features, generate a candidate box in the target features, and obtain features in the candidate box as candidate query features. A second detection module is configured to perform target detection processing on the attention features, generate a candidate box in the attention features, and obtain features in the candidate box as candidate support features. A target object determination module is configured to determine the target object in the target image based on the matching relationship between the candidate query features and the candidate support features.

8. An image detection model training apparatus, characterized by comprising: The device comprises: A sample image acquisition module is configured to acquire a sample image and a sample support image, wherein the sample support image contains a target object, and the sample image contains a target annotation region. A sample extraction module is configured to perform feature extraction on the sample image and the sample support image respectively through a feature extraction module in an image detection model, and obtain sample features and sample support features correspondingly. A sample attention feature generation module is configured to generate a sample attention feature based on the similarity between the sample features and the sample support features. A first sample detection module is configured to perform target detection processing on the sample features through a target detection module in the image detection model, generate a candidate box in the sample features, and obtain features in the candidate box as sample candidate query features. A second sample detection module is configured to perform target detection processing on the sample attention features through the target detection module, generate a candidate box in the sample attention features, and obtain features in the candidate box as sample candidate support features. A region prediction module is configured to obtain the matching relationship between the candidate query features and the candidate support features through a multi-head selection module in the image detection model, and output a predicted object region in the sample image based on the matching relationship. A model training module is configured to train the image detection model according to the predicted object region and the target annotation region in the sample image, so as to detect a target image through the trained image detection model.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the target image detection method according to any one of claims 1 to 4. Alternatively, the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the image detection model training method according to any one of claims 5 or 6.

10. A computer readable storage medium characterized by The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the target image detection method according to any one of claims 1 to 4. Alternatively, the at least one instruction is loaded and executed by the processor to implement the image detection model training method according to any one of claims 5 or 6.

Images