Image recognition method, device, computer equipment and readable storage medium

Abstract

The embodiment of the present application discloses an image recognition method, apparatus, computer equipment and readable storage medium, which are related to computer vision technology of artificial intelligence; specifically, an image to be recognized can be obtained; attribute region detection is performed on the object in the image to be recognized to obtain multiple original attribute regions; the original attribute region is scaled to obtain candidate attribute regions of multiple scales; feature extraction is performed on the candidate attribute region to obtain a multi-scale feature group, and the multi-scale feature group includes the regional features of the candidate attribute region of each scale; based on the multi-scale feature group, the attribute region of the corresponding scale is selected from the candidate attribute regions of multiple scales to obtain a target attribute region combination corresponding to the original attribute region; according to the regional features of the attribute region in the target attribute region combination, the object in the image to be recognized is recognized to obtain a recognition result. This solution can improve the accuracy of image recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image recognition method, device, computer equipment and readable storage medium. Background Art

[0002] The main task of fine-grained image recognition is to identify or divide different subcategories of images belonging to the same large category, such as further classifying bird pictures.

[0003] At present, the commonly used fine-grained image recognition solution is to use artificial intelligence (AI) to achieve image recognition, fine-grained image recognition, etc. Specifically, AI is used to capture the local detail information of objects in the image, and recognize the objects in the image based on the local detail information. For example, a convolutional neural network with image recognition function can be trained using sample images with attribute areas marked, and the image to be recognized is input into the trained convolutional neural network. The convolutional neural network will detect the attribute area of ​​the target object in the image to be recognized, and then the convolutional neural network classifies the target object in the image to be recognized according to the feature information of the attribute area, thereby realizing image recognition.

[0004] Due to the different shooting distances and angles of current images, there will be various differences in scale and other forms between images. However, the current AI-based image recognition solutions cannot adapt to such differences between images. Therefore, the detected attribute areas often have inaccurate positioning problems such as scale and position, resulting in low accuracy of image recognition (or classification). Summary of the invention

[0005] The embodiments of the present application provide an image recognition method, apparatus, computer device, and readable storage medium, which can improve the accuracy of image recognition.

[0006] The present application provides an image recognition method, including:

[0007] Obtain an image to be recognized;

[0008] Performing attribute region detection on the object in the to-be-recognized image to obtain a plurality of original attribute regions;

[0009] Performing a scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales;

[0010] Extracting features from the candidate attribute region to obtain a multi-scale feature group, wherein the multi-scale feature group includes regional features of the candidate attribute region at each scale;

[0011] Based on the multi-scale feature group, an attribute region of a corresponding scale is selected from candidate attribute regions of multiple scales to obtain a target attribute region combination corresponding to the original attribute region;

[0012] According to the regional features of the attribute regions in the target attribute region combination, the object in the image to be identified is identified to obtain a recognition result.

[0013] Accordingly, the embodiment of the present application further provides an image recognition device, including:

[0014] An acquisition unit, used for acquiring an image to be recognized;

[0015] A detection unit, used for performing attribute region detection on the object in the to-be-recognized image to obtain a plurality of original attribute regions;

[0016] A transformation unit, used for performing a scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales;

[0017] A feature extraction unit, configured to extract features from the candidate attribute region to obtain a multi-scale feature group, wherein the multi-scale feature group includes regional features of the candidate attribute region at each scale;

[0018] A region selection unit, configured to select an attribute region of a corresponding scale from candidate attribute regions of multiple scales based on the multi-scale feature group, to obtain a target attribute region combination corresponding to the original attribute region;

[0019] The recognition unit is used to recognize the object in the image to be recognized according to the regional features of the attribute regions in the target attribute region combination to obtain a recognition result.

[0020] In one embodiment, the region selection unit includes:

[0021] A matrix construction subunit, used for constructing a corresponding feature circulant matrix based on the multi-scale feature group;

[0022] A response value calculation subunit, used for calculating a scale response value corresponding to a candidate attribute region combination based on a characteristic circulant matrix, wherein the scale response value is a response value when the scale combination of the candidate attribute region combination is an optimal scale combination;

[0023] The selection subunit is used to select a target attribute region combination corresponding to the original attribute region from the candidate attribute region combination based on the scale response value.

[0024] In one embodiment, the response value calculation subunit is used to: obtain a scale selection weight parameter; and calculate the scale response value of the candidate attribute region combination according to the scale selection weight parameter and the characteristic circulation matrix.

[0025] In one embodiment, the response value calculation subunit is used to: perform Fourier transform processing according to the scale selection weight parameter and the characteristic circulant matrix to obtain the scale response value of the candidate attribute region combination.

[0026] In one embodiment, the identification unit includes:

[0027] A fusion subunit, used to fuse the regional features of the attribute regions in the target attribute region combination to obtain the target region features corresponding to the original attribute regions;

[0028] The recognition subunit is used to recognize the object in the image to be recognized according to the target area feature corresponding to the original attribute area, and obtain a recognition result.

[0029] In one embodiment, the fusion subunit is used to fuse the regional features of the attribute areas in the target attribute area combination corresponding to each target attribute area combination of the original attribute area to obtain the combined regional features corresponding to each target attribute area combination; and obtain the target area features corresponding to the original attribute area according to the combined regional features corresponding to each target attribute area combination.

[0030] In one embodiment, the recognition subunit is used to: perform multiple spatial directional pooling processes on the target area features corresponding to the original attribute area to obtain multiple pooled features; select corresponding features from the multiple pooled features to obtain the target pooled features corresponding to the original attribute area; and identify the object in the image to be identified based on the target pooled features corresponding to the original attribute area to obtain a recognition result.

[0031] In one embodiment, the recognition subunit is used to: fuse the target area features corresponding to the original attribute areas to obtain fused target area features; and recognize the object in the image to be recognized based on the fused target area features to obtain a recognition result.

[0032] In one embodiment, the identification subunit is used to:

[0033] Dividing the target area feature map corresponding to the original attribute area into a plurality of overlapping feature sub-blocks;

[0034] Expand each feature sub-block into a feature vector to obtain a feature vector set corresponding to the feature map;

[0035] The feature vector set is subjected to multiple spatial directional pooling processes to obtain multiple pooled features.

[0036] In one embodiment, the identification subunit is used to:

[0037] Based on the shared fully connected layer, each feature information after pooling is fully connected to obtain feature information with the same dimensions in multiple dimensions.

[0038] Select corresponding features from feature information with the same dimensions to obtain target pooling features corresponding to the attribute region combination.

[0039] In one embodiment, the detection unit is used to perform attribute region detection on the object in the image to be identified using a trained region detection network.

[0040] In one embodiment, the image recognition device may further include: a training unit;

[0041] The training unit is used to:

[0042] Obtain multiple sample images with annotated attribute areas;

[0043] Using a preset region detection network to perform attribute region detection on sample objects in the sample image respectively, to obtain n sample attribute regions of the sample image, where n is a positive integer greater than 1;

[0044] Obtain the cross entropy loss of the s-th sample attribute region in each sample image, where s is a positive integer and is less than or equal to n;

[0045] Obtain similarity information between the s-th sample attribute regions in each sample image;

[0046] The preset region detection network is trained according to the cross entropy loss and the similarity information to obtain a trained region detection network.

[0047] In one embodiment, the transformation unit is used to: determine the center point and multiple vertices of the original attribute area; determine a first vertex far from the center point and a second vertex close to the center point from the multiple vertices; enlarge the scale of the original attribute area according to the first vertex, and reduce the scale of the original attribute area according to the second vertex, to obtain candidate attribute areas of multiple scales.

[0048] Correspondingly, the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the image recognition methods provided in the embodiments of the present application are implemented.

[0049] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps in any one of the image recognition methods provided in the embodiments of the present application are implemented.

[0050] The embodiment of the present application can obtain an image to be identified; perform attribute region detection on the object in the image to be identified to obtain multiple original attribute regions; perform scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales; perform feature extraction on the candidate attribute region to obtain a multi-scale feature group, the multi-scale feature group includes the regional features of the candidate attribute region at each scale; based on the multi-scale feature group, select the attribute region of the corresponding scale from the candidate attribute regions of multiple scales to obtain a target attribute region combination corresponding to the original attribute region; identify the object in the image to be identified based on the regional features of the attribute region in the target attribute region combination to obtain an identification result. This solution can generate multi-scale candidate attribute regions for a single original attribute region, and adaptively select the attribute region of the best scale. Therefore, it can adaptively adapt to images of different scales to achieve accurate positioning or detection of attribute regions, thereby improving the accuracy of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 is a schematic diagram of a scene of an image recognition method provided in an embodiment of the present application;

[0053] Figure 2a is a flow chart of an image recognition method provided by an embodiment of the present application;

[0054] Figure 2b It is a schematic diagram of the framework of image recognition provided by the embodiment of the present application;

[0055] Figure 2c is a schematic diagram of the structure of the DSM module provided in an embodiment of the present application;

[0056] Figure 3a is a bird sample image provided in an embodiment of the present application;

[0057] Figure 3b is another flowchart of the image recognition method provided by an embodiment of the present application;

[0058] Figure 4a is a schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0059] Figure 4b is another structural schematic diagram of the image recognition device provided in an embodiment of the present application;

[0060] Figure 4c is another structural schematic diagram of the image recognition device provided in an embodiment of the present application;

[0061] Figure 4d is another structural schematic diagram of the image recognition device provided in an embodiment of the present application;

[0062] Figure 5 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0064] The embodiments of the present application provide an image recognition method, apparatus, computer device and computer-readable storage medium. The image recognition apparatus can be integrated in a computer device, which can be a server or a terminal.

[0065] The image recognition solution provided in the embodiment of the present application involves computer vision technology (CV) of artificial intelligence. Image recognition such as fine-grained image recognition or classification can be implemented by computer vision technology of artificial intelligence to obtain recognition results.

[0066] Among them, Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, detect and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image segmentation, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0067] In the embodiments of the present application, the so-called image recognition refers to the technology and process of identifying regions with unique properties from an image and finding objects of interest. In the embodiments of the present application, it mainly refers to identifying target objects in an image, for example, identifying the animal type of an animal in an animal image.

[0068] For example, see Figure 1 Taking the image recognition device integrated in a computing device as an example, the computing device can obtain an image to be recognized, for example, the computing device can collect the image with recognition by itself through a camera or other devices; perform attribute region detection on the object in the image to be recognized, and obtain multiple original attribute regions; perform scale transformation on the original attribute region, and obtain candidate attribute regions of multiple scales; perform feature extraction on the candidate attribute region, and obtain a multi-scale feature group, which includes the regional features of the candidate attribute region of each scale; based on the multi-scale feature group, select the attribute region of the corresponding scale from the candidate attribute region of multiple scales, and obtain the target attribute region combination corresponding to the original attribute region; recognize the object in the image to be recognized according to the regional features of the attribute region in the target attribute region combination, and obtain the recognition result. For example, the type of the object in the image can be recognized.

[0069] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0070] This embodiment will be described from the perspective of an image recognition device, which can be specifically integrated into a computer device, which can be a server or a terminal. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro-processing box, or other devices.

[0071] like Figure 2a As shown, the specific process of the image recognition method can be as follows:

[0072] 201. Obtain an image to be recognized.

[0073] The image to be identified is an image containing an object to be identified, and the identification object may be an animal, a person, a building or the like. For example, the image to be identified may be an image of a bird or an image of a circuit board (including defective circuits or devices).

[0074] In the embodiment of the present application, the image to be recognized can be acquired by an image acquisition device and then provided to the image recognition device. The image acquisition device may include a terminal with an image shooting function, such as a mobile phone.

[0075] In addition, in one embodiment, the image to be recognized may also be acquired by an image recognition device, such as a computer device where the image recognition device is located.

[0076] 202. Perform attribute region detection on the object in the image to be identified to obtain multiple original attribute regions.

[0077] The attribute region may be a local region in the image that can represent a certain attribute of the target object (e.g., head), such as a local rectangular region. For example, it may be a local region representing the head, feet, or abdomen of a bird in a bird image. For another example, it may be a local region representing a defect in a circuit board image.

[0078] In the embodiment of the present application, a deep learning network can be used to implement attribute region detection. For example, a region detection network can be trained, and the trained region detection network can be used to detect attribute regions. That is, the step of "performing attribute region detection on objects in the image to be identified" can include: using the trained region detection network to perform attribute region detection on objects in the image to be identified.

[0079] There are many ways to train the region detection network, for example, back propagation can be used for training. For example, sample images with attribute regions labeled can be used to train the region detection network, such as calculating the loss between the predicted attribute region and the labeled attribute region of the region detection network, and training the network based on the loss.

[0080] In one embodiment, in order to improve the detection accuracy of attribute regions and thus improve the accuracy of image recognition, morphological constraints can be introduced during network training so that the i-th attribute regions of different samples have the same semantic information. For example, the first attribute region is a bird's head, thereby achieving semantic alignment of the attribute regions.

[0081] The specific training process is as follows:

[0082] Obtain multiple sample images with annotated attribute areas;

[0083] Using a preset region detection network to perform attribute region detection on sample objects in the sample image, respectively, to obtain n sample attribute regions of the sample image, where n is a positive integer greater than 1;

[0084] Obtain the cross entropy loss of the s-th sample attribute region in each sample image, where s is a positive integer and is less than or equal to n;

[0085] Obtain similarity information between the s-th sample attribute regions in each sample image;

[0086] The preset region detection network is trained according to the cross entropy loss and similarity information to obtain the trained region detection network.

[0087] The cross entropy may be the cross entropy between the predicted s-th sample attribute region and the labeled s-th attribute region, and may be calculated based on the labeled region and the predicted region.

[0088] The similarity information may include Euclidean distance, etc., and may be calculated based on the regional features of the sth sample attribute regions in each sample image.

[0089] For example, the embodiment of the present application can introduce morphological constraints when training the region detection network to achieve semantic alignment of attribute regions (for example, making the sth attribute regions of different samples correspond to the head). The loss function containing the constraints is expressed as: The first two terms of the loss function represent the cross entropy loss of the sth attribute region of two samples i and j, and the last term represents the Euclidean distance of the sth attribute region features of two samples i and j. The embodiment of the present application can achieve semantic alignment of attribute regions by constraining the distance of the sth attribute regions of different samples, improve the accuracy of attribute region detection and positioning, and thus improve the accuracy of image classification (image recognition).

[0090] Through the above steps, original attribute regions, such as the head region of the bird, the abdomen region of the bird, the foot region of the bird, etc., can be detected from the image to be identified, such as the bird image.

[0091] 203. Perform a scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales.

[0092] For example, a scale transformation may be performed on each original attribute region to obtain candidate attribute regions of multiple scales (multi-scale candidate attribute regions) corresponding to each original attribute region.

[0093] Since the scales of different images are different, in order to be able to adapt to the scales of various images, accurately locate the attribute area, and improve the accuracy of image recognition, the embodiment of the present application can generate multi-scale candidate attribute areas for a single attribute area, and then adaptively select the best scale from the multi-scale candidate attribute areas to achieve alignment of the multi-scale attribute areas and accurately locate the attribute areas.

[0094] There are many ways of scale transformation, such as reduction, enlargement, lengthening, etc. The specific method can be selected according to actual needs. The number of candidate attribute regions generated can be set according to actual needs, such as an odd number of regions.

[0095] In one embodiment, the original attribute region is reduced and enlarged to obtain a multi-scale candidate attribute region of the original attribute region. Specifically, the scaling process of the original attribute region is as follows:

[0096] Determine the center point and multiple vertices of the original attribute area;

[0097] Determine a first vertex far from the center point and a second vertex close to the center point from the plurality of vertices;

[0098] The scale of the original attribute region is enlarged according to the first vertex, and the scale of the original attribute region is reduced according to the second vertex, so as to obtain candidate attribute regions of multiple scales.

[0099] For example, the center point and two vertices of the original attribute area are determined, and then, according to a preset rule, one of the vertices is randomly defined as a first vertex far from the center point and the other vertex is defined as a second vertex close to the center point.

[0100] In one embodiment, the first vertex may be fixed, and the scale of the original attribute region may be enlarged based on the scale scaling factor; the second vertex may be fixed, and the scale of the original attribute region may be reduced based on the scale scaling factor.

[0101] For example, in one embodiment, reference Figure 2b , is a frame image of image recognition in the embodiment of the present application, such as Figure 2b As shown in the figure, multiple bird images to be identified can be obtained, and then the original attribute region in each bird image can be detected through the region detection network. Then, the scale mining discriminative module (DSM) can be used to implement scale transformation, optimal region selection and other operations, thereby realizing the alignment of multi-scale attribute regions and the precise positioning of attribute regions.

[0102] Among them, the DSM module can generate multi-scale attribute regions and realize the alignment and response value output of multi-scale attribute regions. Assume that the DSM module uses A i =[φ i , ψ i , η i , γ i ] represents the i-th attribute region, where φ i represents the attribute region feature, ψ i and η i Represents the coordinates of two vertices in the attribute area, γ i =(w i ,h i ) represents the width and height of the attribute area.

[0103] First, calculate the center coordinates of the selected K attribute regions and set ψ i Defined as the vertex away from the center, η i Defined as the vertex close to the center. The algorithm generates S (odd number) multi-scale candidate attribute regions by zooming in and out on the original attribute region. '

[0104] If the attribute area is reduced, η should be fixed i , reduce the width and height of the original attribute area to get candidate attribute regions, represented by

[0105] If the attribute area is enlarged, ψ should be fixed i , enlarge the width and height of the original attribute area to get candidate attribute regions, represented by Where α is the scaling factor.

[0106] Through the above steps, a multi-scale candidate attribute region may be generated for each original attribute region, for example, a multi-scale candidate attribute region corresponding to the bird's head region.

[0107] 204. Perform feature extraction on the candidate attribute region to obtain a multi-scale feature group, where the multi-scale feature group includes regional features of the candidate attribute region at each scale.

[0108] For example, feature extraction can be performed on the candidate attribute region of each scale of the original attribute to obtain a multi-scale feature group.

[0109] In one embodiment, in order to improve the efficiency and accuracy of feature extraction, a convolutional neural network may be used to extract features of the candidate attribute regions, such as using an image feature extraction network, which may be a residual network.

[0110] For example, refer to Figure 2b , the convolutional neural network in the DSM module can be used to extract the regional features of each candidate attribute region of the original attribute region to obtain a set of multi-scale features

[0111] 205. Based on the multi-scale feature group, select an attribute region of a corresponding scale from candidate attribute regions of multiple scales to obtain a target attribute region combination corresponding to the original attribute region.

[0112] For example, for each original attribute region, a corresponding number of attribute regions may be selected from the multi-scale candidate attribute regions corresponding to each original attribute region to obtain an attribute region combination corresponding to each original attribute region.

[0113] The target attribute region combination includes one or more attribute regions selected from the multi-scale candidate attribute regions. For example, candidate attribute regions of three scales may be selected from five candidate attribute regions of the bird head region to form the attribute region combination.

[0114] In one embodiment, the target attribute region combination may be an attribute region combination that is optimal for the scale of the current image to be recognized, that is, for the scale of the current recognized image, the scale of the region in the target attribute region combination is the most matching or optimal scale.

[0115] In one embodiment, in order to improve the selection efficiency and accuracy of the attribute region, multi-scale alignment can be achieved through a circulant matrix expression (Toeplitz matrix) based on multi-scale features and a multi-scale response value can be obtained, and the attribute region combination can be selected based on the scale response value. For example, the step of "selecting attribute regions of corresponding scales from candidate attribute regions of multiple scales based on the multi-scale feature group to obtain a target attribute region combination corresponding to the original attribute region" may include:

[0116] Constructing corresponding feature circulant matrix based on multi-scale feature groups;

[0117] Calculate the scale response value corresponding to the candidate attribute region combination based on the characteristic circulant matrix, the scale response value is the response value used to characterize that the scale combination of the candidate attribute region combination is the optimal scale combination;

[0118] A target attribute region combination corresponding to the original attribute region is selected from the candidate attribute region combinations based on the scale response value.

[0119] The candidate attribute region combination may include a region combination consisting of a predetermined number of candidate attribute regions in the multi-scale candidate attribute regions.

[0120] The predetermined number is the number of regions to be selected from the multi-scale candidate attribute regions, which can be set according to actual needs, such as 1, 3, 4, etc. For example, the candidate attribute region combination can be a region combination consisting of any three candidate attribute regions in the multi-scale candidate attribute regions.

[0121] Among them, the scale response value corresponding to the candidate attribute region combination is the response value of the scale combination of the candidate attribute region combination as the optimal scale combination; the scale response value represents the distance between the candidate attribute region combination and the optimal scale combination, etc.; the larger the response value, the smaller the distance from the optimal scale combination (that is, the closer to the optimal scale combination); conversely, the smaller the response value, the greater the distance from the optimal scale combination (that is, the farther away from the optimal scale combination).

[0122] In the embodiment of the present application, there are multiple candidate attribute region combinations, and the number can be equal to the number of combinations of the predetermined number of regions selected from the multi-scale candidate attribute regions. In the embodiment of the present application, the scale response value of each or each candidate attribute region combination can be calculated based on the characteristic circulant matrix, and then the attribute region combination corresponding to the original attribute region (i.e., the optimal attribute region combination) can be determined from the multiple candidate attribute region combinations according to the scale response value.

[0123] In one embodiment, the attribute region combination may be selected by setting a scale selection weight parameter; specifically, the step of “calculating the scale response value corresponding to the candidate attribute region combination based on the characteristic circulant matrix” may include:

[0124] Get the scale selection weight parameter;

[0125] According to the scale selection weight parameter and the characteristic circulation matrix, the scale response value of the candidate attribute region combination is calculated.

[0126] For example, in one embodiment, the scale selection weight parameter and the feature circulation matrix may be matrix multiplied by matrix product to obtain the scale response value of the candidate attribute region combination.

[0127] For example, refer to Figure 2b , the convolutional neural network in the DSM module is used to extract the regional features of each candidate attribute region of the original attribute region, and a set of multi-scale features is obtained Afterwards, a set of multi-scale features can be constructed The circulant matrix is ​​as follows:

[0128]

[0129] in, Represents the feature vector The kth element of .

[0130] Among them, the response value of the candidate attribute region combination can be calculated by the following formula:

[0131] in This is the weight parameter selected for the scale.

[0132] It can be seen that the embodiment of the present application can realize the selection of region combinations through the cyclic shift of the cyclic matrix, which can improve the efficiency and accuracy of the region combination selection, and further improve the positioning accuracy of the attribute region.

[0133] In one embodiment, in order to improve the calculation speed of the response value and improve the image recognition efficiency, the response value can be calculated by fast Fourier transform (FFT). That is, the step of "calculating the scale response value of the candidate attribute region combination according to the scale selection weight parameter and the characteristic circulant matrix" can include: performing Fourier transform processing according to the scale selection weight parameter and the characteristic circulant matrix to obtain the scale response value of the candidate attribute region combination.

[0134] For example, according to the characteristics of the circulant matrix, the response value can also be calculated by FFT:

[0135] in represents the Fourier matrix, (·) C represents complex conjugate, the corresponding algorithm time complexity is O(KSlog 2 S ).

[0136] In the back propagation phase of the DSM module, if δ i Represents y i The error, then The gradient of can be expressed as in yes Corresponding to the shifted expression. The gradient of can be expressed as Therefore, the response value prediction method based on the cyclic shift expression of multi-scale features can achieve adaptive selection of the optimal scale.

[0137] Through the above steps, the best target attribute region combination can be selected from the multi-scale candidate attribute regions of the original attribute region. Figure 2c , which is the structural diagram of the DSM module. The original attribute region (i.e., rectangular box) in the input bird image is detected by the region detection network. Then, DSM can be used to perform scale transformation on each original attribute region to obtain 5 scale candidate attribute regions for each original attribute region. Then, the regional features of each scale candidate attribute region are extracted based on the convolutional neural network, and the circulant matrix module is used to select 3 best attribute regions from the 5 scale candidate attribute regions to obtain the best region combination.

[0138] In one embodiment, there may be multiple optimal attribute region combinations, so multiple attribute region combinations may be selected from the candidate attribute region combinations based on the scale response value, and then a target attribute region combination, i.e., the final optimal region combination, may be selected from the multiple attribute region combinations. There are multiple ways to select the target attribute region combination from the multiple attribute region combinations, for example, the combination may be selected based on the feature value corresponding to the attribute region combination, for example, the combination with the largest feature value may be selected as the target attribute region combination, and so on.

[0139] In practical applications, after selecting a target attribute region combination of an original attribute region, any region within the target attribute region combination may be used to represent the combination, or the combination itself may be directly used to represent the combination.

[0140] For example, refer to Figure 2b ,After detecting the original attribute regions of the two bird images, the DSM module can be used to ,select the best attribute region (e.g. the best attribute region combination) that matches the scale of the bird ,image.

[0141] 206. According to the regional features of the attribute regions in the target attribute region combination, the object in the image to be identified is identified to obtain a recognition result.

[0142] Specifically, the object in the image to be identified may be identified according to the regional features of the attribute regions in the target attribute region combination corresponding to each original attribute region.

[0143] For example, Figure 2b As shown, the bird in the image to be identified can be identified by using the regional features in the target attribute region combination corresponding to each part of the bird (head, abdomen, feet, etc.).

[0144] In one embodiment, in order to improve the efficiency and accuracy of recognition, the regional features in each target attribute region combination may be fused. Specifically, the step of “recognizing the object in the image to be recognized according to the regional features of the attribute region in the target attribute region combination to obtain a recognition result” may include:

[0145] The regional features of the attribute regions in the target attribute region combination are fused to obtain the target region features corresponding to the original attribute regions;

[0146] According to the target area features corresponding to the original attribute area, the object in the image to be identified is identified to obtain the identification result.

[0147] For example, for each target attribute region combination corresponding to the original attribute region, the regional features of the attribute regions in the target attribute region combination are fused to obtain the combined region features corresponding to each target attribute region combination; the target region features corresponding to the original attribute region are obtained based on the combined region features corresponding to each target attribute region combination; the object in the image to be identified is identified based on the target region features to obtain the identification result.

[0148] For example, the selection may be based on the size of the characteristic values ​​of the combined regional features of each target attribute region, such as selecting the combined regional feature with the largest characteristic value as the target regional feature of the original attribute region.

[0149] For example, reference Figure 2c After the DSM module selects multiple optimal attribute region combinations (including 3 optimal attribute regions) of the original attribute region, the region features of the 3 attribute regions in the region combination can be fused to obtain the combined region features corresponding to each optimal attribute region combination. Then, the combined region feature with the largest feature value is selected from the combined region features of each optimal attribute region combination as the target region feature of the original attribute region for subsequent image classification or recognition.

[0150] In one embodiment, considering that due to the target image recognition scheme, global pooling is used in the convolutional neural network to extract the features of the attribute region, the spatial directionality information of the attribute region is lost, and the attribute region cannot be spatially oriented and aligned, resulting in low accuracy of image classification or recognition.

[0151] To overcome the aforementioned problems, in the embodiments of the present application, directional pooling processing can be used to achieve spatial directional alignment of attribute regions. For example, for two bird head regions of the same category facing left and right, similar features can be extracted, thereby improving the accuracy of image classification or recognition.

[0152] Specifically, the step of “identifying the object in the image to be identified according to the target area features corresponding to the original attribute area to obtain the identification result” may include:

[0153] Perform multiple spatial directional pooling processes on the target area features corresponding to the original attribute area to obtain multiple pooled features;

[0154] Select corresponding features from multiple pooled features to obtain target pooled features corresponding to the original attribute area;

[0155] According to the target pooling features corresponding to the original attribute area, the object in the image to be identified is identified to obtain the recognition result.

[0156] Among them, there are many spatial directionalities, such as horizontal, vertical, rotational horizontal, rotational vertical, etc.

[0157] In one embodiment, in order to facilitate the spatial directional pooling operation and improve efficiency, the regional feature map corresponding to the original attribute area can be divided and combined. Specifically, the step of "performing multiple spatial directional pooling processes on the target area features corresponding to the original attribute area to obtain multiple pooled features" may include:

[0158] Divide the target area feature map corresponding to the original attribute area into multiple overlapping feature sub-blocks;

[0159] Expand each feature sub-block into a feature vector to obtain a set of feature vectors corresponding to the feature map;

[0160] The feature vector set is pooled in multiple spatial directions to obtain multiple pooled features.

[0161] For example, refer to Figure 2bThe global average pooling operation commonly used in the field of image recognition will lose a lot of spatial information. This technology proposes a directional pooling (OP) module to achieve spatial directional alignment of image features, which helps to extract common features of images with directional differences such as symmetry, rotation, and translation.

[0162] After obtaining the feature map of the original attribute area After that, the OP module first divides the feature map into N H ×N W There are overlapping sub-blocks, and the height and width of each sub-block are h×w respectively.

[0163] Then the features of each sub-block are expanded into a vector, so the feature map can be re-expressed as Next, the OP module uses horizontal pooling, vertical pooling, and rotated horizontal pooling and vertical pooling on the re-expressed feature map. The four pooling operations can be expressed by the following formula:

[0164]

[0165]

[0166]

[0167]

[0168] After obtaining multiple pooled features, the best pooled feature can be selected from them. For example, the step of "selecting corresponding features from multiple pooled features to obtain target pooled features corresponding to the original attribute area" may include:

[0169] Based on the shared fully connected layer, each feature information after pooling is fully connected to obtain feature information with the same dimensions in multiple dimensions.

[0170] Select corresponding features from feature information with the same dimensions to obtain target pooling features corresponding to the attribute region combination.

[0171] For example, after the OP module undergoes the above four directional pooling, the OP module selects the best one from the four pooling methods by taking the maximum value to obtain the final response value output, that is, y=max(ω T Γ h ,ω T Γ v ,ω T Γ′ h ,ω T Γ′ v), where ω represents the parameter of the shared fully connected layer. For images with different perspectives, this directional pooling module can better achieve feature alignment and retain more spatial information, further improving the accuracy of classification or recognition.

[0172] After obtaining the target region features corresponding to each original attribute region, the embodiment of the present application can identify the object in the image based on the target region features corresponding to each original attribute region. In order to improve the recognition accuracy, in one embodiment, the features of each region can be fused, and the fused features can be used to identify the object. Specifically, the step of "identifying the object in the image to be identified according to the target region features corresponding to the original attribute region, and obtaining the recognition result" may include:

[0173] The target region features corresponding to each original attribute region are fused to obtain the fused target region features;

[0174] The object in the image to be identified is identified according to the fused target area features to obtain the recognition result.

[0175] There are many ways to fuse the target area features. For example, weighted summation can be used for fusion; for example, according to the feature weights corresponding to the target area, weighted summation is performed on the target area features corresponding to each original attribute area.

[0176] For example, refer to Figure 2b First, the attribute area of ​​the bird image is accurately located using the DSM module, and then the spatial feature alignment of the obtained attribute area is realized using the OP module. Finally, the classification result is predicted by weighted summation of the features of different attribute areas. Specifically, after obtaining the regional features of each part of the bird image through the DSM and OP modules, the weight parameters such as feature weights w1, w2, and w3 can be used to continue weighted summation of the regional features of each part of the bird image. Then, the type of bird in the bird image, such as sparrow, can be identified based on the summed features.

[0177] The image recognition method provided in the embodiment of the present application can be applied to any image recognition scenario. For example, in the current development of the chip and circuit board industry, the demand for fault detection is very large. Traditional defect detection requires a large number of experienced experts to carry out classification, which has high personnel requirements, high costs and low efficiency. Most of these defects appear as roughly similar black areas in appearance, but differ in details, such as some defects with halos, and some defects are similar to water stains. Therefore, the image fine-grained classification algorithm provided in the embodiment of the present application can be used for identification. At the same time, the defects in the image show the characteristics of different sizes and asymmetric spatial morphology, which are suitable for identification using space and scale alignment algorithms.

[0178] For example, the image to be identified may be a circuit board image containing a defective area. Traditional circuit defects can mostly appear as black areas on the image. Therefore, the defective area in the circuit image may be a black area. According to the above description, the present application is implemented by first using the DSM module to accurately locate the attribute area of ​​the circuit image (the attribute area may be an area characterizing the regionality of the defect), and then using the OP module to achieve spatial feature alignment of the obtained attribute area, and finally achieving the prediction of the classification result by weighted summation of the features of different attribute areas.

[0179] Specifically, please refer to Figure 2b The DSM module detects the original attribute region (i.e., rectangular box) in the input circuit image through the region detection network, and then uses DSM to perform scale transformation on each original attribute region to obtain multiple scale candidate attribute regions for each original attribute region; then, the regional features of each scale candidate attribute region are extracted based on the convolutional neural network, and a predetermined number of optimal attribute regions are selected from the multiple scale candidate attribute regions through the circulant matrix module to obtain the optimal region combination.

[0180] You can refer to Figure 2c After the DSM module selects multiple optimal attribute region combinations (including 3 optimal attribute regions) of the original attribute region, the regional features of the attribute regions in the region combinations can be fused to obtain the combined regional features corresponding to each optimal attribute region combination. Then, the combined regional features with the largest feature value are selected from the combined regional features of each optimal attribute region combination as the target regional features of the original attribute region for subsequent image classification or recognition.

[0181] The OP module performs multiple spatial directional pooling processes on the target area features, such as the four types mentioned above, to obtain the pooled area features of each original attribute area, and uses the weight parameter to perform weighted summation on the pooled area features of each original attribute area. Then, based on the summed features, the type of defective area in the circuit image is identified, such as burnout, damage, etc.

[0182] In addition, the method of the embodiment of the present application can also be applied to many areas such as mobile phone model identification in the field of e-commerce, and has broad application prospects. For specific application implementation, please refer to the introduction of the above embodiment.

[0183] As can be seen from the above, the embodiment of the present application obtains an image to be identified; performs attribute region detection on an object in the image to be identified to obtain multiple original attribute regions; performs scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales; performs feature extraction on the candidate attribute region to obtain a multi-scale feature group, the multi-scale feature group includes regional features of the candidate attribute region of each scale; based on the multi-scale feature group, selects attribute regions of corresponding scales from the candidate attribute regions of multiple scales to obtain a target attribute region combination corresponding to the original attribute region; identifies the object in the image to be identified based on the regional features of the attribute regions in the target attribute region combination to obtain a recognition result. This solution can generate multi-scale candidate attribute regions for a single original attribute region, and adaptively select the attribute region of the best scale. Therefore, it can adaptively adapt to images of different scales to achieve accurate positioning or detection of attribute regions, thereby improving the accuracy of image recognition.

[0184] In addition, the scheme can also introduce morphological constraints to allow the i-th attribute regions of different samples to have the same semantic information head. After achieving semantic alignment of the attribute regions, a directional pooling algorithm is used to achieve spatial directional alignment of the attribute regions. For example, for two bird head regions facing left and right of the same category, similar features are extracted to further improve the accuracy of classification or image recognition.

[0185] The method described in the above embodiments is further described in detail below with examples.

[0186] In this embodiment, the image recognition device is specifically integrated into a computer device as an example for description.

[0187] Currently, there are public dataset samples for fine-grained image classification (reference Figure 3a ) analysis shows that images have various forms of differences due to different shooting distances and shooting angles. For example, the upper and lower images in each column of images in the figure belong to the same category, but the images in the first column on the left have scale differences, the images in the second column on the left have symmetry differences, the images in the third column on the right have rotation differences, and the images in the first column on the right have translation differences. Therefore, it is necessary to design an algorithm to accurately locate the attribute area of ​​the image and realize the spatial alignment of the attribute features to further improve the accuracy of classification or image recognition.

[0188] In order to improve the accuracy of image recognition or classification, the embodiments of the present application may adopt the following solutions:

[0189] (A) Network training.

[0190] First, the computer device can obtain a sample image set and train the image recognition network based on the sample image set. Figure 2bThe image recognition network may include: a region detection network, a DSM (scale mining and discrimination) module, and an OP module. The structure of the DSM module may refer to the description of the above embodiment.

[0191] In the embodiment of the present application, the image recognition network can be trained based on sample images with object types labeled, such as by back propagation. The training method of the region detection network can refer to the above introduction.

[0192] (ii) After training, the image recognition network can recognize the target object in the image to be recognized.

[0193] like Figure 3b As shown, an image recognition method, the specific process is as follows:

[0194] 301. A computer device obtains an image to be recognized.

[0195] For example, a computer device may receive images of birds uploaded by a user.

[0196] 302. The computer device uses a region detection network to perform attribute region detection on the object in the image to be identified, and obtains multiple original attribute regions.

[0197] For example, a computer device can input a bird image to be identified into a region detection network in an image recognition network, and detect the original attribute regions of the bird image, such as the bird's head region, foot region, abdomen region, etc., through the region detection network.

[0198] For another example, the computer device can input the defective circuit board image to be identified into the region detection network in the image recognition network, and detect the original attribute region of the defective circuit board image through the region detection network.

[0199] 303. The computer device performs scale transformation on each original attribute region through a scale mining and discrimination module to obtain a multi-scale candidate attribute region for each original attribute region.

[0200] For example, the DSM module may generate a multi-scale attribute region by scaling. For specific scaling methods, reference may be made to the description of the above embodiment.

[0201] 304. The computer device extracts features from the candidate attribute regions through a scale mining and discrimination module to obtain a multi-scale feature group for each original attribute region.

[0202] Among them, the multi-scale feature group includes the regional features of the candidate attribute regions at each scale

[0203] For example, the DSM module can extract regional features through a convolutional network.

[0204] 305. The computer device constructs a feature circulant matrix corresponding to the multi-scale feature group through a scale mining discrimination module.

[0205] For example, the convolutional neural network in the DSM module is used to extract the regional features of each candidate attribute region of the original attribute region to obtain a set of multi-scale features. Afterwards, a set of multi-scale features can be constructed The circulant matrix of . For the specific circulant matrix, refer to the above description.

[0206] 306. The computer device calculates the scale response value of the candidate attribute region combination based on the scale selection weight parameter and the feature circulation matrix through the scale mining discrimination module.

[0207] Among them, the scale response value corresponding to the candidate attribute region combination is the response value of the candidate attribute region combination as the optimal scale combination; the scale response value represents the distance between the candidate attribute region combination and the optimal scale combination; the larger the response value, the smaller the distance from the optimal scale combination (i.e., the closer to the optimal scale combination); conversely, the smaller the response value, the larger the distance from the optimal scale combination (i.e., the farther away from the optimal scale combination). For the specific calculation of the scale response value, please refer to the above introduction.

[0208] 307. The computer device selects a target attribute region combination corresponding to the original attribute region from the candidate attribute region combinations based on the scale response value through the scale mining discrimination module.

[0209] Specifically, a combination of attribute regions whose response values ​​are ranked in the top N may be selected, where N is a positive integer greater than 1.

[0210] 308. The computer device fuses the regional features of the attribute regions in the target attribute region combination through the scale mining and discrimination module to obtain the target region features corresponding to the original attribute regions.

[0211] For example, the computer device can fuse the regional features of the attribute areas in the target attribute area combination corresponding to each target attribute area combination of the original attribute area to obtain the combined regional features corresponding to each target attribute area combination; and obtain the target area features corresponding to the original attribute area according to the combined regional features corresponding to each target attribute area combination.

[0212] 309. The computer device performs multiple spatial directional pooling processes on the target area features corresponding to the original attribute area through a directional pooling module to obtain multiple pooled features.

[0213] For the specific pooling method, please refer to the description of the above embodiment.

[0214] 310. The computer device selects corresponding features from multiple pooled features through a directional pooling module to obtain a target pooling feature corresponding to the original attribute area.

[0215] For example, a full connection operation can be performed on the pooled features to obtain feature information with the same dimensions in multiple dimensions; corresponding features can be selected from the feature information with the same dimensions in multiple dimensions to obtain the target pooled features corresponding to the attribute region combination.

[0216] 311. The computer device inputs the target pooling features into the classification network, classifies the objects in the image to be identified through the classification network, and obtains the identification results.

[0217] For example, the classification network classifies the target object in the image based on the input target pooling features, obtains the classification result, and uses the classification result as the recognition result.

[0218] For example, when the image to be identified is a bird image, the bird in the bird image can be identified as a yellow siskin in the above manner.

[0219] For another example, when the image to be identified is an image of a defective circuit board, the type of the circuit board defect can be identified as damage or the like through the above method.

[0220] From the above, it can be seen that the embodiment of the present application can generate multi-scale candidate attribute regions for a single original attribute region through a scale mining and discrimination module, and adaptively select the attribute region of the best scale. Therefore, it can adapt to images of different scales to achieve accurate positioning or detection of attribute regions, thereby improving the accuracy of image recognition.

[0221] In addition, the scheme can also introduce morphological constraints to allow the i-th attribute regions of different samples to have the same semantic information head. After achieving semantic alignment of the attribute regions, the directional pooling module uses a variety of directional pooling algorithms to achieve spatial directional alignment of the attribute regions. For example, for the two left and right bird head regions of the same category, similar features are extracted to further improve the accuracy of classification or image recognition.

[0222] In order to better implement the above method, the embodiment of the present application further provides an image recognition device, which can be integrated into a computer device, such as a server or a terminal.

[0223] For example, Figure 4a As shown, the image recognition device may include an acquisition unit 401, a detection unit 402, a transformation unit 403, a feature extraction unit 404, a region selection unit 405, and a recognition unit 406, as follows:

[0224] An acquisition unit 401 is used to acquire an image to be recognized;

[0225] A detection unit 402 is used to perform attribute region detection on the object in the image to be identified to obtain a plurality of original attribute regions;

[0226] A transformation unit 403, configured to perform a scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales;

[0227] A feature extraction unit 404 is used to extract features from the candidate attribute region to obtain a multi-scale feature group, wherein the multi-scale feature group includes regional features of the candidate attribute region at each scale;

[0228] A region selection unit 405 is used to select an attribute region of a corresponding scale from candidate attribute regions of multiple scales based on the multi-scale feature group, so as to obtain a target attribute region combination corresponding to the original attribute region;

[0229] The recognition unit 406 is used to recognize the object in the image to be recognized according to the region features of the attribute regions in the target attribute region combination to obtain a recognition result.

[0230] In one embodiment, reference Figure 4b , the area selection unit 405 comprises:

[0231] A matrix construction subunit 4051, configured to construct a corresponding feature circulant matrix based on the multi-scale feature group;

[0232] A response value calculation subunit 4052 is used to calculate the scale response value corresponding to the candidate attribute region combination based on the characteristic circulant matrix, wherein the scale response value is the response value when the scale combination of the candidate attribute region combination is the optimal scale combination;

[0233] The selection subunit 4053 is used to select a target attribute region combination corresponding to the original attribute region from the candidate attribute region combination based on the scale response value.

[0234] In one embodiment, the response value calculation subunit 4053 is used to: obtain a scale selection weight parameter; and calculate the scale response value of the candidate attribute region combination according to the scale selection weight parameter and the characteristic circulation matrix.

[0235] In one embodiment, the response value calculation subunit 4053 is used to: perform Fourier transform processing according to the scale selection weight parameter and the characteristic circulant matrix to obtain the scale response value of the candidate attribute region combination.

[0236] In one embodiment, reference Figure 4c , the identification unit 406 may include:

[0237] A fusion subunit 4061 is used to fuse the regional features of the attribute regions in the target attribute region combination to obtain the target region features corresponding to the original attribute regions;

[0238] The recognition subunit 4062 is used to recognize the object in the image to be recognized according to the target area feature corresponding to the original attribute area to obtain a recognition result.

[0239] In one embodiment, the fusion subunit 4061 is used to fuse the regional features of the attribute areas in the target attribute area combination for each target attribute area combination corresponding to the original attribute area, so as to obtain the combined regional features corresponding to each target attribute area combination; and obtain the target area features corresponding to the original attribute area according to the combined regional features corresponding to each target attribute area combination.

[0240] In one embodiment, the recognition subunit 4062 is used to: perform multiple spatial directional pooling processes on the target area features corresponding to the original attribute area to obtain multiple pooled features; select corresponding features from the multiple pooled features to obtain the target pooled features corresponding to the original attribute area; and identify the object in the image to be identified based on the target pooled features corresponding to the original attribute area to obtain a recognition result.

[0241] In one embodiment, the recognition subunit 4062 is used to: fuse the target area features corresponding to the original attribute areas to obtain fused target area features; and recognize the object in the image to be recognized based on the fused target area features to obtain a recognition result.

[0242] In one embodiment, the identification subunit 4062 is used to:

[0243] Dividing the target area feature map corresponding to the original attribute area into a plurality of overlapping feature sub-blocks;

[0244] Expand each feature sub-block into a feature vector to obtain a feature vector set corresponding to the feature map;

[0245] The feature vector set is subjected to multiple spatial directional pooling processes to obtain multiple pooled features.

[0246] In one embodiment, the identification subunit 4062 is used to:

[0247] Based on the shared fully connected layer, each feature information after pooling is fully connected to obtain feature information with the same dimensions in multiple dimensions.

[0248] Select corresponding features from feature information with the same dimensions to obtain target pooling features corresponding to the attribute region combination.

[0249] In one embodiment, the detection unit 402 is used to use a trained region detection network to perform attribute region detection on the object in the image to be identified.

[0250] In one embodiment, reference Figure 4d , the image recognition device may further include: a training unit 407;

[0251] The training unit is used to:

[0252] Obtain multiple sample images with annotated attribute areas;

[0253] Using a preset region detection network to perform attribute region detection on sample objects in the sample image respectively, to obtain n sample attribute regions of the sample image, where n is a positive integer greater than 1;

[0254] Obtain the cross entropy loss of the s-th sample attribute region in each sample image, where s is a positive integer and is less than or equal to n;

[0255] Obtain similarity information between the s-th sample attribute regions in each sample image;

[0256] The preset region detection network is trained according to the cross entropy loss and the similarity information to obtain a trained region detection network.

[0257] In one embodiment, the transformation unit 403 is used to: determine the center point and multiple vertices of the original attribute area; determine a first vertex far from the center point and a second vertex close to the center point from the multiple vertices; enlarge the scale of the original attribute area according to the first vertex, and reduce the scale of the original attribute area according to the second vertex, to obtain candidate attribute areas of multiple scales.

[0258] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0259] As can be seen from the above, the embodiment of the present application can be used in an image recognition device to obtain an image to be recognized through an acquisition unit 401; the detection unit 402 performs attribute region detection on the object in the image to be recognized to obtain multiple original attribute regions; the transformation unit 403 performs scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales; the feature extraction unit 404 performs feature extraction on the candidate attribute region to obtain a multi-scale feature group, and the multi-scale feature group includes the regional features of the candidate attribute region of each scale; the region selection unit 405 selects the attribute region of the corresponding scale from the candidate attribute regions of multiple scales based on the multi-scale feature group to obtain a target attribute region combination corresponding to the original attribute region; the recognition unit 406 recognizes the object in the image to be recognized according to the regional features of the attribute region in the target attribute region combination to obtain a recognition result. This solution can generate multi-scale candidate attribute regions for a single original attribute region, and adaptively select the attribute region of the best scale, so it can adaptively adapt to images of different scales to achieve accurate positioning or detection of attribute regions, thereby improving the accuracy of image recognition.

[0260] The present application also provides a computer device, such as Figure 5 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0261] The computer device may include one or more processing core processors 501, one or more computer-readable storage media memories 502, a power supply 503, an input unit 504 and other components. Those skilled in the art will appreciate that Figure 5 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0262] The processor 501 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 502 and calling data stored in the memory 502, it executes various functions of the computer device and processes data, thereby performing overall detection of the computer device. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0263] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0264] The computer device also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 503 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0265] The computer device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0266] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 501 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502, thereby realizing various functions, as follows:

[0267] Acquire an image to be identified; perform attribute region detection on an object in the image to be identified to obtain a plurality of original attribute regions; perform scale transformation on the original attribute region to obtain candidate attribute regions of a plurality of scales; perform feature extraction on the candidate attribute region to obtain a multi-scale feature group, wherein the multi-scale feature group includes regional features of the candidate attribute region of each scale; based on the multi-scale feature group, select attribute regions of a corresponding scale from the candidate attribute regions of a plurality of scales to obtain a target attribute region combination corresponding to the original attribute region; identify the object in the image to be identified according to the regional features of the attribute regions in the target attribute region combination to obtain an identification result.

[0268] For details of the above operations, please refer to the previous embodiments, which will not be described in detail here.

[0269] As can be seen from the above, after acquiring the image to be identified, the computer device of this embodiment performs attribute region detection on the object in the image to be identified to obtain multiple original attribute regions; performs scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales; performs feature extraction on the candidate attribute region to obtain a multi-scale feature group, and the multi-scale feature group includes the regional features of the candidate attribute region at each scale; based on the multi-scale feature group, selects the attribute region of the corresponding scale from the candidate attribute regions of multiple scales to obtain the target attribute region combination corresponding to the original attribute region; and identifies the object in the image to be identified according to the regional features of the attribute region in the target attribute region combination to obtain the identification result. This solution can generate multi-scale candidate attribute regions for a single original attribute region, and adaptively select the attribute region of the best scale. Therefore, it can adaptively adapt to images of different scales to achieve accurate positioning or detection of attribute regions, thereby improving the accuracy of image recognition.

[0270] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0271] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any medical image segmentation method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0272] Acquire an image to be identified; perform attribute region detection on an object in the image to be identified to obtain a plurality of original attribute regions; perform scale transformation on the original attribute region to obtain candidate attribute regions of a plurality of scales; perform feature extraction on the candidate attribute region to obtain a multi-scale feature group, wherein the multi-scale feature group includes regional features of the candidate attribute region of each scale; based on the multi-scale feature group, select attribute regions of a corresponding scale from the candidate attribute regions of a plurality of scales to obtain a target attribute region combination corresponding to the original attribute region; identify the object in the image to be identified according to the regional features of the attribute regions in the target attribute region combination to obtain an identification result.

[0273] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0274] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0275] Since the instructions stored in the computer-readable storage medium can execute the steps in any image recognition method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image recognition method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0276] The above is a detailed introduction to an image recognition method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image recognition method, characterized in that: include: Obtain an image to be recognized; Performing attribute region detection on the object in the to-be-recognized image to obtain a plurality of original attribute regions; Performing a scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales; Extracting features from the candidate attribute region to obtain a multi-scale feature group, wherein the multi-scale feature group includes regional features of the candidate attribute region at each scale; Based on the multi-scale feature group, an attribute region of a corresponding scale is selected from candidate attribute regions of multiple scales to obtain a target attribute region combination corresponding to the original attribute region, wherein the step of selecting an attribute region of a corresponding scale from candidate attribute regions of multiple scales based on the multi-scale feature group to obtain a target attribute region combination corresponding to the original attribute region includes: Constructing a corresponding feature circulant matrix based on the multi-scale feature group; calculating a scale response value corresponding to a candidate attribute region combination based on the feature circulant matrix, wherein the scale response value is a response value of the scale combination of the candidate attribute region combination as an optimal scale combination; selecting a target attribute region combination corresponding to an original attribute region from the candidate attribute region combination based on the scale response value; The step of selecting a target attribute region combination corresponding to the original attribute region from the candidate attribute region combination based on the scale response value comprises: selecting a plurality of attribute region combinations from the candidate attribute region combination based on the scale response value; and taking the attribute region combination with the largest characteristic value among the plurality of attribute region combinations as the target attribute region combination; According to the regional features of the attribute regions in the target attribute region combination, the object in the image to be identified is identified to obtain a recognition result.

2. The image recognition method according to claim 1, characterized in that: The scale response value corresponding to the candidate attribute region combination is calculated based on the characteristic circulant matrix, including: Get the scale selection weight parameter; The scale response value of the candidate attribute region combination is calculated according to the scale selection weight parameter and the characteristic circulation matrix.

3. The image recognition method according to claim 2, characterized in that: Calculating the scale response value of the candidate attribute region combination according to the scale selection weight parameter and the characteristic circulation matrix includes: Fourier transform processing is performed according to the scale selection weight parameter and the characteristic circulant matrix to obtain the scale response value of the candidate attribute region combination.

4. The image recognition method according to claim 1, characterized in that: According to the regional features of the attribute regions in the target attribute region combination, the object in the image to be identified is identified to obtain the identification result, including: The regional features of the attribute regions in the target attribute region combination are fused to obtain the target region features corresponding to the original attribute regions; According to the target area features corresponding to the original attribute area, the object in the image to be identified is identified to obtain a recognition result.

5. The image recognition method according to claim 4, characterized in that: The regional features of the attribute regions in the target attribute region combination are fused to obtain the target region features corresponding to the original attribute regions, including: For each target attribute region combination corresponding to the original attribute region, regional features of the attribute regions in the target attribute region combination are fused to obtain a combined regional feature corresponding to each target attribute region combination; According to the combined region features corresponding to each target attribute region combination, the target region features corresponding to the original attribute region are obtained.

6. The image recognition method according to claim 4, characterized in that: According to the target area features corresponding to the original attribute area, the object in the image to be identified is identified to obtain the identification result, including: Performing multiple spatial directional pooling processes on the target area features corresponding to the original attribute area to obtain multiple pooled features; Selecting corresponding features from the multiple pooled features to obtain target pooled features corresponding to the original attribute area; According to the target pooling features corresponding to the original attribute area, the object in the image to be identified is identified to obtain a recognition result.

7. The image recognition method according to claim 4, characterized in that: According to the target area feature corresponding to the original attribute area, the object in the image to be identified is identified to obtain the identification result, and the identification result includes: The target region features corresponding to each original attribute region are fused to obtain the fused target region features; The object in the image to be identified is identified according to the fused target area features to obtain a recognition result.

8. The image recognition method according to claim 6, characterized in that: The target area features corresponding to the original attribute area are subjected to multiple spatial directional pooling processes to obtain multiple pooled features, including: Dividing the target area feature map corresponding to the original attribute area into a plurality of overlapping feature sub-blocks; Expand each feature sub-block into a feature vector to obtain a feature vector set corresponding to the feature map; The feature vector set is subjected to multiple spatial directional pooling processes to obtain multiple pooled features.

9. The image recognition method according to claim 6, characterized in that: Selecting corresponding features from the multiple pooled features to obtain target pooled features corresponding to the original attribute area includes: Based on the shared fully connected layer, each feature information after pooling is fully connected to obtain feature information with the same dimensions in multiple dimensions. Select corresponding features from feature information with the same dimensions to obtain target pooling features corresponding to the attribute region combination.

10. The image recognition method according to claim 1, characterized in that: Performing attribute region detection on the object in the to-be-recognized image includes: using a trained region detection network to perform attribute region detection on the object in the to-be-recognized image.

11. The image recognition method according to claim 10, characterized in that: Also includes: Obtain multiple sample images with annotated attribute areas; Using a preset region detection network to perform attribute region detection on sample objects in the sample image respectively, to obtain n sample attribute regions of the sample image, where n is a positive integer greater than 1; Obtain the cross entropy loss of the s-th sample attribute region in each sample image, where s is a positive integer and is less than or equal to n; Obtain similarity information between the s-th sample attribute regions in each sample image; The preset region detection network is trained according to the cross entropy loss and the similarity information to obtain a trained region detection network.

12. An image recognition device, characterized in that: include: An acquisition unit, used for acquiring an image to be recognized; A detection unit, used for performing attribute region detection on the object in the to-be-recognized image to obtain a plurality of original attribute regions; A transformation unit, used for performing a scale transformation on the original attribute region to obtain candidate attribute regions of multiple scales; A feature extraction unit, configured to extract features from the candidate attribute region to obtain a multi-scale feature group, wherein the multi-scale feature group includes regional features of the candidate attribute region at each scale; A region selection unit is used to select an attribute region of a corresponding scale from candidate attribute regions of multiple scales based on the multi-scale feature group to obtain a target attribute region combination corresponding to the original attribute region; the region selection unit is specifically used to: construct a corresponding feature circulant matrix based on the multi-scale feature group; Calculate the scale response value corresponding to the candidate attribute region combination based on the characteristic circulant matrix, wherein the scale response value is the response value when the scale combination of the candidate attribute region combination is the optimal scale combination; Selecting a target attribute region combination corresponding to the original attribute region from the candidate attribute region combination based on the scale response value; The step of selecting a target attribute region combination corresponding to the original attribute region from the candidate attribute region combination based on the scale response value comprises: selecting a plurality of attribute region combinations from the candidate attribute region combination based on the scale response value; and taking the attribute region combination with the largest characteristic value among the plurality of attribute region combinations as the target attribute region combination; The recognition unit is used to recognize the object in the image to be recognized according to the regional features of the attribute regions in the target attribute region combination to obtain a recognition result.

13. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 11 are implemented.

Images