Image recognition method, model training method, electronic device, medium, and product

Abstract

An image recognition method, an image recognition model training method, an electronic device, a computer-readable storage medium, and a computer program product, which can be applied to the fields of computer technology, deep learning technology, and image processing technology. The image recognition method comprises: acquiring an image to be recognized (S210); splitting a feature map of said image into a plurality of feature map patches (S220); performing attention feature extraction on the plurality of feature map patches to obtain a channel-enhanced feature map (S230); performing prompt recognition on said image to obtain spatial prompt information (S240); and performing semantic segmentation on the channel-enhanced feature map on the basis of the spatial prompt information to obtain a recognition map for a target object (S250).

Description

Image recognition method, training method of image recognition model, electronic device, medium and product TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer technology, deep learning technology and image processing technology, and more particularly, to an image recognition method, a training method of an image recognition model, an electronic device, a storage medium and a program product. BACKGROUND

[0002] Machine vision technology refers to a technology that uses image processing technology to realize the recognition, detection, measurement and control of objects. With the development of artificial intelligence technology, machine vision technology is increasingly widely used in industry. For example, in the electronics industry, machine vision technology can be used to detect defects in products to improve product pass rates and reduce labor costs. However, how to efficiently and accurately detect products has become a research focus.

[0003] SUMMARY

[0004] The present disclosure provides an image recognition method, a training method of an image recognition model, an image recognition device, a training device of an image recognition model, an electronic device, a computer-readable storage medium and a computer program product.

[0005] According to one aspect of the present disclosure, an image recognition method is provided, comprising: obtaining a to-be-recognized image; splitting a feature map of the to-be-recognized image to obtain a plurality of block feature map blocks; performing attention feature extraction on the plurality of block feature map blocks to obtain a channel-enhanced feature map; performing prompt recognition on the to-be-recognized image to obtain spatial prompt information; and performing semantic segmentation on the channel-enhanced feature map based on the spatial prompt information to obtain a recognition map for a target object.

[0006] According to another aspect of the present disclosure, a training method of an image recognition model is provided, comprising: splitting a sample feature map of a sample input image to obtain a plurality of sample feature map blocks; performing attention feature extraction on the plurality of sample feature map blocks to obtain a channel-enhanced sample feature map; performing prompt recognition on the sample input image to obtain sample spatial prompt information; performing semantic segmentation on the channel-enhanced sample feature map based on the sample spatial prompt information to obtain a sample recognition map for a target object; classifying the sample recognition map to obtain a sample recognition result; and training the image recognition model based on the sample recognition result and a sample label corresponding to the sample input image, wherein the sample label indicates a class result of the target object.

[0007] According to another aspect of the present disclosure, an image recognition device is provided, comprising: an acquisition module configured to acquire an image to be recognized; a splitting module configured to split a feature map of the image to be recognized to obtain a plurality of feature map blocks; an extraction module configured to perform attention feature extraction on the plurality of feature map blocks to obtain a channel-enhanced feature map; a prompt recognition module configured to perform prompt recognition on the image to be recognized to obtain spatial prompt information; and a segmentation module configured to perform semantic segmentation on the channel-enhanced feature map based on the spatial prompt information to obtain a recognition map for a target object.

[0008] According to another aspect of the present disclosure, a training device for an image recognition model is provided, comprising: a sample splitting module configured to split a sample feature map of a sample input image to obtain a plurality of sample feature map blocks; a sample extraction module configured to perform attention feature extraction on the plurality of sample feature map blocks to obtain a channel-enhanced sample feature map; a sample prompt module configured to perform prompt recognition on the sample input image to obtain sample spatial prompt information; a sample segmentation module configured to perform semantic segmentation on the channel-enhanced sample feature map based on the sample spatial prompt information to obtain a sample recognition map for a target object; a sample classification module configured to classify the sample recognition map to obtain a sample recognition result; and a sample training module configured to train the image recognition model based on the sample recognition result and a sample label corresponding to the sample input image, wherein the sample label indicates a category result of the target object. According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory configured to store one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement a method as described in the present disclosure.

[0009] According to another aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon executable instructions that, when executed by a processor, cause the processor to implement a method as described in the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions for implementing a method as described in the present disclosure when executed. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0012] FIG. 1A schematically shows an application scenario of an image recognition method according to an embodiment of the present disclosure;

[0013] FIG. 1B schematically shows an application scenario of the image recognition method according to another embodiment of the present disclosure;

[0014] FIG. 2 schematically shows a flowchart of the image recognition method according to an embodiment of the present disclosure;

[0015] FIG. 3 schematically shows a diagram of obtaining a plurality of feature map blocks according to an embodiment of the present disclosure;

[0016] FIG. 4 schematically shows a flowchart of attention feature extraction according to an embodiment of the present disclosure;

[0017] FIG. 5A schematically shows a diagram of global attention attention feature extraction according to an embodiment of the present disclosure;

[0018] FIG. 5B schematically shows a diagram of local attention attention feature extraction according to another embodiment of the present disclosure;

[0019] FIG. 6 schematically shows a network structure diagram of a feature extraction module according to an embodiment of the present disclosure;

[0020] FIG. 7A schematically shows a diagram of determining region of interest information according to an embodiment of the present disclosure;

[0021] FIG. 7B schematically shows a diagram of determining a target pixel according to an embodiment of the present disclosure;

[0022] FIG. 8A schematically shows a network structure diagram of a semantic segmentation module according to a related example;

[0023] FIG. 8B schematically shows a network structure diagram of a semantic segmentation module according to another embodiment of the present disclosure;

[0024] FIG. 9 schematically shows a network structure diagram of an adapter according to an embodiment of the present disclosure;

[0025] FIG. 10 schematically shows a network structure diagram of an optimizer according to an embodiment of the present disclosure;

[0026] FIG. 11A schematically shows a diagram of an image recognition model according to an embodiment of the present disclosure;

[0027] FIG. 11B schematically shows a diagram of an image recognition model according to another embodiment of the present disclosure;

[0028] FIG. 12 schematically shows a diagram of a to-be-recognized image and a task feature enhanced feature map according to an embodiment of the present disclosure;

[0029] FIG. 13 schematically shows a flowchart of a training method of an image recognition model according to an embodiment of the present disclosure;

[0030] FIG. 14 schematically illustrates a matrix decomposition schematic diagram according to an embodiment of the present disclosure;

[0031] FIG. 15 schematically illustrates a flow schematic diagram of a training method of an image recognition model according to an embodiment of the present disclosure;

[0032] FIG. 16 schematically illustrates a block diagram of an electronic device adapted to implement an image recognition method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description which follows is merely illustrative and is not intended to limit the scope of the present disclosure. In the following detailed description of embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it would be apparent to one skilled in the art that one or more embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present disclosure.

[0034] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprise" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0035] All terms used herein, including technical and scientific terms, have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings that are consistent with the context of the present description, and should not be interpreted in an idealized or excessively formal manner.

[0036] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include at least one of each item enumerated, but not to exclude others not enumerated. For example, "a system having at least one of A, B, and C" should be interpreted to include systems having at least one of A, B, or C, etc., but not to exclude systems having two or more of A, B, and C, etc.

[0037] In the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, user equipment information such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, public order and good customs are not violated, and corresponding operation portals are provided for the user to choose authorization or refusal.

[0038] In the scenario of making automated decisions using personal information, the method, device and system provided by the embodiments of the present application all provide corresponding operation portals for the user to choose to agree or refuse the automated decision result; if the user chooses to refuse, the expert decision process is entered. The expression "automated decision" here refers to the activity of automatically analyzing, evaluating the behavior habits, interests and hobbies or economic, health, credit status of individuals, etc. by computer programs and making decisions. The expression "expert decision" here refers to the activity of making decisions by personnel who are engaged in a certain field of work, have specialized experience, knowledge and skills and have reached a certain professional level.

[0039] Image preprocessing and deep learning model design and development technology is one of the most important breakthroughs in the field of artificial intelligence in recent years. It has achieved great success in computer vision, image and video analysis, multimedia, machine vision defect detection and many other fields. Deep learning model design and development technology is a very important research direction in computer vision tasks, and as an urgent problem to be solved in industry, it has many application needs in industry.

[0040] In a computer vision task example, a processor in an electronic device can receive a to-be-identified image about a target item collected from a camera, classify and identify whether a target object exists in the target item in the to-be-identified image, to determine whether the target item in the to-be-identified image is abnormal. The target item can include a product. The target object can include a crack or other defects. For example, the processor can be configured with a conventional deep learning model, such as a convolutional model, to identify the to-be-identified image using the conventional deep learning model to obtain an identification result representing whether the target object exists in the target item in the to-be-identified image. A sorter in the electronic device sorts the target item with the target object.

[0041] However, when the light is dim or the target item is in a backlight environment, the to-be-identified image collected by the camera can have problems such as large noise or low image resolution, which can result in poor accuracy of the identification result obtained by processing the to-be-identified image using the conventional deep learning model, and the normal target item without the target object can be easily sorted out.

[0042] The present disclosure provides an image recognition method, a training method of an image recognition model, a training device of an image recognition model, an electronic device, a computer readable storage medium and a computer program product, which can be applied to the technical fields of computer technology, deep learning technology and image processing technology. The image recognition method comprises: obtaining a to-be-recognized image; splitting a feature map of the to-be-recognized image to obtain a plurality of block feature map blocks; performing attention feature extraction on the plurality of block feature map blocks to obtain a channel-enhanced feature map; performing prompt recognition on the to-be-recognized image to obtain spatial prompt information; and performing semantic segmentation on the channel-enhanced feature map based on the spatial prompt information to obtain a recognition map for a target object.

[0043] FIG. 1A schematically shows an application scenario diagram of an image recognition method according to an embodiment of the present disclosure.

[0044] As shown in FIG. 1A, the image recognition method can be applied in a sorting scene. The application scenario 100 can include a camera 110, a processor 120 and a sorter 130. The camera 110, the processor 120 and the sorter 130 can communicate through a network.

[0045] The camera 110 is configured to collect image information of a target object 140 to obtain a to-be-recognized image. The camera 110 can include a camera, a mobile phone, a computer or other electronic devices with a camera module. The camera 110 can send the collected to-be-recognized image about the target object 140 to the processor 120, so that the processor 120 analyzes the to-be-recognized image to obtain a recognition map of whether the target object 140 exists. Based on the recognition map, a recognition result of whether the target object 140 exists is obtained. The processor 120 can send the recognition result to the sorter 130. In the case that the recognition result represents that the target object 140 exists, the sorter 130 can sort the target object.

[0046] For example, the processor 120 can be configured to split a feature map of the to-be-recognized image to obtain a plurality of feature map blocks. Attention feature extraction is performed on the plurality of feature map blocks to obtain a channel-enhanced feature map. Semantic segmentation is performed on the channel-enhanced feature map to obtain a recognition map for a target object. Based on the recognition map, a recognition result of whether the target object 140 exists in the to-be-recognized image is obtained.

[0047] The sorter 130 is configured to sort the target object 140 in the case that the recognition result represents that the target object 140 exists. The structure of the sorter 130 is not limited, for example, it can include a mechanical hand which can grab the abnormal object and sort it out from the queue of normal objects.

[0048] In an example, the target object can refer to a product. For example, in a production workshop, a product that has been produced is subjected to a quality inspection, and a to-be-identified image about the product is acquired by a camera. The camera can send the to-be-identified image to a processor, so that the processor analyzes the to-be-identified image to obtain an identification result. The identification result can represent whether the product is a defective product, such as a product with a crack. In a case where the identification result represents that the product is a defective product, the product can be sorted out by a sorter.

[0049] FIG. 1B schematically shows an application scenario of an image recognition method according to another embodiment of the present disclosure.

[0050] Unlike the sorting scenario shown in FIG. 1A, the sorting scenario 100 shown in FIG. 1B includes multiple cameras 111, 112, and 113, and further includes a decision maker 160. The multiple cameras 111, 112, and 113 are respectively arranged at different positions of a target object 150 or a conveying belt for conveying the target object 150, and multiple processors 121, 122, and 123 are respectively loaded in the different cameras 111, 112, and 113.

[0051] For example, the multiple cameras 111, 112, and 113 loaded with the processors respectively can acquire to-be-identified images about the same target object 150 from different positions. The to-be-identified images are acquired by the loaded processors. The to-be-identified images are processed to obtain an identification image of the target object. Based on the identification image of the target object, an identification result is obtained, which represents whether the target object exists in the target object. The decision maker 160 is configured to acquire the identification results respectively determined by the multiple processors 121, 122, and 123. The decision maker 160 determines a target identification result based on the multiple identification results. The decision maker 160 is further configured to send the target identification result to the sorter 130, so that the sorter 130 sorts out the target object 150 in a case where the target identification result represents that the target object 150 exists in the target object 150.

[0052] For example, the decision maker 160 is configured to acquire the identification images respectively determined by the multiple processors 121, 122, and 123. In a case where the identification result represents that the target object exists in the target object, the identification image is taken as the target identification image. The decision maker 160 only needs to receive information of the processor that determines the target identification image. The decision maker 160 sends a result represented by the target identification image that the target object exists in the target object to the sorter 130, so that the sorter 130 sorts out the target object 150. In the following, the image recognition method provided by the embodiments of the present disclosure will be described in detail based on the sorting scenarios shown in FIG. 1A and FIG. 1B.

[0053] It should be noted that the serial numbers of the various operations in the following method are only used to represent the operations for description, and should not be regarded as representing the execution order of the various operations. The method does not need to be executed in the order shown unless explicitly indicated.

[0054] FIG. 2 schematically shows a flowchart of an image recognition method according to an embodiment of the present disclosure.

[0055] As shown in FIG. 2, the image recognition method of this embodiment includes operation S210 to operation S250.

[0056] In operation S210, an image to be recognized is obtained.

[0057] In operation S220, a feature map of the image to be recognized is split to obtain a plurality of feature map blocks.

[0058] In operation S230, attention feature extraction is performed on the plurality of feature map blocks to obtain a channel-enhanced feature map.

[0059] In operation S240, prompt recognition is performed on the image to be recognized to obtain spatial prompt information.

[0060] In operation S250, based on the spatial prompt information, semantic segmentation is performed on the channel-enhanced feature map to obtain a recognition map for a target object.

[0061] The image to be recognized can be obtained from a camera. The camera obtains image information about a target object. The category of the target object is not limited, which can be a product to be detected, but is not limited thereto, and can be other organisms such as plants, etc. As long as the image to be recognized includes image information of the target object.

[0062] The feature map can be obtained by vector conversion of the image to be recognized. For example, the image to be recognized is converted into a feature map through an image embedding operation. The image to be recognized can be processed by a 16*16 convolution layer with a step size of 16 to convert the image to be recognized into a high-dimensional feature map, but is not limited thereto, and a convolution layer and a normalization layer can also be used for vector conversion of the image to be recognized to obtain a feature map, as long as a model network that can achieve the purpose of vector conversion is used. The feature map can also be obtained by feature extraction or encoding of the image to be recognized. For example, the image to be recognized can be processed by using a long short-term memory network (LSTM) and / or an encoder in a transformer to obtain a feature map.

[0063] Optionally, the to-be-recognized image can be pre-processed, and the pre-processed to-be-recognized image is converted into a vector to obtain a feature map. The pre-processing can include binarization processing, but is not limited thereto, and can also include cropping. The image region including the target object can be cropped by target detection to obtain the pre-processed to-be-recognized image. Through the pre-processing operation, the amount of information of unnecessary image information in the feature map can be reduced, thereby reducing the data amount of processing the feature map, improving the processing efficiency, and improving the processing accuracy.

[0064] The feature map block can be obtained by splitting the feature map, and the splitting manner can include channel dimension splitting or spatial dimension splitting and channel dimension splitting. For example, the feature map is split in the channel dimension. The feature map includes C channels, and c channels are taken as a splitting unit, and then C / c feature map blocks can be obtained. For another example, the feature map can be split in the channel dimension and the spatial dimension. The execution order of the channel dimension splitting and the spatial dimension splitting is not limited.

[0065] The attention feature extraction can include feature extraction or feature encoding in an attention mechanism. The attention mechanism can include a self-attention mechanism, but is not limited thereto, and can also include a cross-attention mechanism, or can include a combination of the attention mechanism and other operations such as convolution, pooling, normalization, and multi-layer perception, as long as the feature extraction manner can improve the attention of the image information related to the segmentation target object. Exemplarily, the encoder in the Transformer (encoder-decoder) can be used to encode the plurality of feature map blocks, thereby realizing attention feature extraction of the plurality of feature map blocks.

[0066] Since the feature map block is obtained by splitting the feature map, the splitting granularity is refined to the channel dimension. The plurality of feature map blocks are processed by using the attention feature extraction manner, and a channel-enhanced feature map is obtained.

[0067] The plurality of feature map blocks can also be processed by using other feature extraction manners to obtain a channel-enhanced feature map. For example, the convolutional network or the long short-term memory network can be used to extract features of the plurality of feature map blocks to obtain a channel-enhanced feature map. Compared with other feature extraction manners, the attention feature extraction manner can fully learn the context semantic relationship between the plurality of feature map blocks, and improve the information richness of the channel-enhanced feature map obtained after feature extraction.

[0068] The spatial prompt information can be understood as image position prompt information. The spatial prompt information can be used as auxiliary information for semantic segmentation. For example, the spatial prompt information can include pixel position information about the target object, but is not limited thereto, and can also include image region information about the region where the target object is located. As long as the prompt information can assist in improving the semantic segmentation accuracy of the channel-enhanced feature map, it is acceptable.

[0069] The spatial prompt information can be obtained by prompting recognition of the to-be-recognized image. For example, the spatial prompt information can be obtained by extracting information for semantic segmentation from the to-be-recognized image. For example, the prompting recognition can be understood as extraction, but is not limited thereto, and can also be understood as screening, conversion, refinement, or determination. For example, the to-be-recognized image includes spatial prompt information and other image information, and the spatial prompt information can be obtained from the to-be-recognized image by prompting recognition.

[0070] The semantic segmentation can include pixel category recognition. The image pixels represented by the channel-enhanced feature map can be assigned to corresponding categories by using the spatial prompt information to obtain a recognition map about the target object. The semantic segmentation can include at least one of encoding, decoding, semantic segmentation feature optimization, and semantic segmentation feature adaptation. The channel-enhanced feature map with rich feature information can be subjected to a task feature-enhanced segmentation operation by semantic segmentation, so that the obtained recognition map for the target object is a pixel information corresponding to the target object, and the pixel information corresponding to non-target objects is a masked segmentation map.

[0071] The semantic segmentation can be directly performed on the channel-enhanced feature map to obtain a recognition map for the target object. However, it is not limited thereto. The semantic segmentation can also be performed on the channel-enhanced feature map based on the spatial prompt information to obtain a recognition map. Since the spatial prompt information includes pixel position information or region position information about the target object in the to-be-recognized image, the segmentation accuracy and effectiveness can be improved by using the spatial prompt information as auxiliary information for the semantic segmentation of the channel-enhanced feature map.

[0072] According to an embodiment of the present disclosure, by splitting the feature map corresponding to the to-be-identified image, the image information of the to-be-identified image is decoupled, the analysis granularity of the feature map can be refined, and thus the attention feature extraction is performed on the image feature information from a fine granularity. In addition, the to-be-identified image is prompted for recognition to obtain spatial prompt information. The channel-enhanced feature map is segmented by using the spatial prompt information to obtain a segmentation map for the target object, and the feature map is further optimized, the effective features related to the object class of the to-be-identified image are enhanced by the spatial prompt information, and the invalid features are weakened, thereby improving the segmentation accuracy and effectiveness of the semantic segmentation. The problem of low image recognition accuracy caused by noise or low resolution of the to-be-identified image taken in a dim or backlit environment is reduced.

[0073] The operation S210 of obtaining the to-be-identified image will be further explained below with reference to FIG. 2.

[0074] According to an embodiment of the present disclosure, for the operation S210 of obtaining the to-be-identified image, a plurality of to-be-identified images of the target object can be obtained. The plurality of to-be-identified images can be obtained from different directions of the target object.

[0075] For example, a camera is arranged at each of different directions of a place where the target object must pass through. The camera is used to collect image information of the target object to obtain a plurality of to-be-identified images.

[0076] The image recognition method shown in FIG. 2 can be performed on each of the plurality of to-be-identified images to obtain a plurality of recognition images corresponding to the plurality of to-be-identified images.

[0077] For example, the recognition images can be classified to obtain a recognition result. The recognition result can represent whether the object class of the target object exists in the to-be-identified image. In a case where the recognition result represents that the object class of the target object exists in the to-be-identified image, the recognition image can be used as a target recognition image. The target recognition image represents that the target object exists in the target object. In a case where the recognition result represents that the object class of the target object does not exist in the to-be-identified image, the analysis on the recognition image can be stopped.

[0078] In a case where at least one target recognition image exists in the plurality of recognition images, it can be determined that the target object exists in the target object. However, it is not limited thereto. In a case where the number of target recognition images is greater than half of the total number of recognition images, it can be determined that the target object exists in the target object. In a case where the number of target recognition images is less than or equal to half of the total number of recognition images, it can be determined that the target object does not exist in the target object.

[0079] In the embodiments of the present disclosure, in a case where the target identification chart characterizes that the target object exists in the target item, that is, the target identification result of the target object existing in the target item is determined, the identification accuracy of the target item with a flaw or a crack can be ensured, and the problem that the target item with a flaw is mistaken for a good product can be avoided.

[0080] The operation S220 of splitting the feature map of the to-be-identified image shown in FIG. 2 will be further described below in combination with FIG. 3.

[0081] FIG. 3 schematically shows a diagram of obtaining a plurality of feature map blocks according to an embodiment of the present disclosure.

[0082] As shown in FIG. 3, the to-be-identified image 310 can be subjected to vector conversion to obtain a feature map 320. The specific explanation of the vector conversion mode can be referred to the explanation of how to obtain the feature map in the example shown in FIG. 2. The feature map 320 can include a three-dimensional matrix, which is schematically shown by a cube in FIG. 3. The length of the feature map 320 is H, the width is W, and the number of channels is C. The face where the length and the width of the feature map 320 are located is defined as a spatial dimension, and the channel dimension is defined as being perpendicular to the spatial dimension.

[0083] As shown in FIG. 3, the feature map 320 can be subjected to spatial dimension splitting. The splitting mode can be referred to the dashed line shown in the feature map 320 in FIG. 3, and the feature map 320 is split into a plurality of sub-feature maps 321. The plurality of sub-feature maps 321 have the same amount of data in the spatial dimension.

[0084] For example, the size of the feature map 320 is H*W*C. The preset split size of the spatial dimension is z, and the feature map is split into 9 non-overlapping sub-feature maps according to the preset split size. The size of each sub-feature map can include z*z*C.

[0085] As shown in FIG. 3, each sub-feature map 321 can be subjected to channel dimension splitting. The splitting mode can be referred to the dashed line shown in the sub-feature map 321 in FIG. 3, and the sub-feature map 321 is split into a plurality of feature map blocks 3211.

[0086] For example, the channels of the sub-feature map 321 include C, and channel c is taken as a splitting unit, and then C / c feature map blocks 3211 can be obtained by splitting.

[0087] It should be noted that the execution order of the spatial dimension splitting and the channel dimension splitting of the feature map is not limited, as long as the feature map can be split in the spatial dimension and the channel dimension.

[0088] The feature maps are split in spatial dimension and channel dimension to obtain a plurality of feature map blocks, so that the feature granularity of the plurality of feature map blocks is fine, and thus the attention feature extraction is performed on the plurality of feature map blocks, which can improve the feature analysis granularity of the attention feature extraction, so that the obtained channel-enhanced feature map is comprehensive and effective.

[0089] The above is explained by combining FIG. 3, and the operation S220 shown in FIG. 2 is explained by splitting the feature map to obtain a plurality of feature map blocks. The operation S230 shown in FIG. 2 is further explained below.

[0090] According to an embodiment of the present disclosure, for the operation S230 shown in FIG. 2, the attention feature extraction is performed on the plurality of feature map blocks to obtain a plurality of channel-enhanced feature map blocks, which can include: performing parallel attention feature extraction on the plurality of feature map blocks to obtain a plurality of channel-enhanced feature map blocks. The plurality of channel-enhanced feature map blocks are combined to obtain a channel-enhanced feature map.

[0091] For example, a plurality of attention heads corresponding to the plurality of feature map blocks can be set, and the plurality of attention heads are used to process the plurality of feature map blocks respectively, so as to realize parallel processing and obtain a plurality of channel-enhanced feature map blocks. Each attention head can use a self-attention mechanism to perform attention feature extraction on the feature map block corresponding to the attention head.

[0092] The plurality of channel-enhanced feature map blocks corresponding to the plurality of sub-feature maps can be combined, which can be single-round combination or multi-round combination, for example, multi-round combination of channel dimension combination and spatial dimension combination, to obtain a channel-enhanced feature map. The execution order of the channel dimension combination and the spatial dimension combination is not limited, as long as the channel-enhanced feature map corresponding to the local information position in the feature map can be obtained.

[0093] The operation S230 shown in FIG. 2 will be further explained below by combining FIG. 4, which is the attention feature extraction operation performed on the plurality of feature map blocks obtained by splitting shown in FIG. 3.

[0094] FIG. 4 schematically shows a flow diagram of attention feature extraction according to an embodiment of the present disclosure.

[0095] As shown in FIG. 4, by performing spatial dimension splitting and channel dimension splitting on the feature map, a plurality of feature map blocks are obtained. Taking the first row and the first column of the sub-feature map 410 obtained by performing spatial dimension splitting on the feature map as an example. By performing channel dimension splitting on the sub-feature map 410, the feature map blocks 411, 412 and 413 are obtained. The feature map block 411 is input into the attention head M411, the feature map block 412 is input into the attention head M412, and the feature map block 413 is input into the attention head M413. The plurality of attention heads are used to process the plurality of feature map blocks in parallel to obtain a plurality of channel-enhanced feature map blocks 511, 512 and 513. The plurality of channel-enhanced feature map blocks are combined to obtain a channel-enhanced sub-feature map corresponding to the sub-feature map 410. The plurality of channel-enhanced sub-feature maps are combined to obtain a channel-enhanced feature map 510.

[0096] According to an embodiment of the present disclosure, in the case that the image to be recognized includes different image information, the plurality of attention heads can be used to perform attention feature extraction on the plurality of feature map blocks in parallel, thereby paying attention to different image information, enriching image features from the channel dimension, and improving the processing efficiency of attention feature extraction by using parallel processing, thereby reducing the time delay problem of subsequent task feature enhancement processing and improving the overall inference efficiency.

[0097] The attention feature extraction of the feature map block by the attention head as shown in FIG. 4 can be global attention, but is not limited thereto. The attention feature extraction of the feature map block by the attention head as shown in FIG. 4 can also be local attention.

[0098] The attention feature extraction of the feature map block by the attention head as shown in FIG. 4 will be further explained below in combination with FIG. 5A.

[0099] According to an embodiment of the present disclosure, the parallel attention feature extraction on the plurality of feature map blocks to obtain the plurality of channel-enhanced feature map blocks can include: performing global attention attention feature extraction on the feature map block based on the position embedding information to obtain the channel-enhanced feature map block.

[0100] The global attention attention feature extraction on the feature map block based on the position embedding information can be understood as learning different types of features included in the feature map block by taking the position embedding information as the prompt information of feature extraction to obtain the global attention channel-enhanced feature map block.

[0101] In an example, the global attention attention feature extraction on the feature map block can include: performing self-attention mechanism calculation on the position embedding information, and the query vector, the key vector and the value vector determined based on the feature map block to obtain the channel-enhanced feature map block.

[0102] The position embedding information indicates that the feature map patch corresponds to an image position in the image to be recognized.

[0103] Determining the query vector, the key vector and the value vector based on the feature map patch can include: performing linear projection on the feature map patch to obtain the query vector Q, the key vector K and the value vector V for attention feature extraction.

[0104] For example, the query vector Q, the key vector K and the value vector V can be calculated by linear projection of formulas (1)-(3).

[0105] wherein Q i represents the query vector of the i-th attention head, x i represents the feature map patch corresponding to the i-th attention head, and respectively represent the learnable linear parameters for determining the query vector of the i-th attention head, K i represents the key vector of the i-th attention head, and respectively represent the learnable linear parameters for determining the key vector of the i-th attention head, V i represents the value vector of the i-th attention head, and respectively represent the learnable linear parameters for determining the value vector of the i-th attention head.

[0106] The channel-enhanced feature map patch can be calculated by the attention mechanism of formula (4).

[0107] wherein Atten i represents the i-th channel-enhanced feature map patch output by the i-th attention head, represents the dot product of the query vector of the i-th attention head and the transpose of the key vector of the i-th attention head, d k represents the dimension of the key vector of the i-th attention head, pos represents the position embedding information, and softmax represents an activation function.

[0108] FIG. 5A schematically shows a schematic diagram of global attention attention feature extraction according to an embodiment of the present disclosure.

[0109] As shown in FIG. 5A, the feature map block 510 can be linearly projected based on different linear projection parameters to obtain a query vector Query 520, a key vector Key 530, and a value vector Value 540. The query vector 520 is contracted, for example, multiplied by a contraction ratio Scale, and the contracted result is dot multiplied with the transpose of the key vector 530. The dot product result is combined with positional embedding information (pos), and the combined result is activated by an activation function 550 to obtain attention weights The attention weights are multiplied with the value vector 540 to obtain the output result of each attention head.

[0110] According to an embodiment of the present disclosure, by performing global attention feature extraction on the feature map block, the information richness of the channel enhanced feature map block can be improved, and the attention degree of the key feature can be improved while avoiding missing image information.

[0111] The feature map block can be subjected to attention feature extraction by using the global attention feature extraction manner, but is not limited thereto, and can also be subjected to attention feature extraction by using a local attention feature extraction manner. The feature map block is subjected to attention feature extraction by using the global attention manner, which is described in detail above, and the feature map block is subjected to attention feature extraction by using the local attention manner, which is described in detail below.

[0112] According to another embodiment of the present disclosure, the parallel attention feature extraction on the plurality of feature map blocks to obtain the plurality of channel enhanced feature map blocks can include: performing local attention feature extraction on the feature map block based on the positional embedding information and the segmentation hint information to obtain the plurality of channel enhanced feature map blocks.

[0113] The positional embedding information indicates the image position of the feature map block corresponding to the image to be recognized, and the segmentation hint information indicates the expected feature type to be extracted. The expected feature type to be extracted can include a feature of an object category used for recognizing an object.

[0114] The local attention feature extraction on the feature map block can be understood as learning different image information included in the feature map block by using the positional embedding information and the segmentation hint information as hint information for feature extraction to obtain the local attention channel enhanced feature map block.

[0115] In an example, the positional embedding information and the segmentation hint information can be concatenated to obtain the hint information. However, it is not limited thereto. The positional embedding information and the segmentation hint information can also be encoded by using a hint encoder to obtain the hint information. The network structure of the hint encoder is not limited, as long as it is a network structure with encoding effect.

[0116] FIG. 5B schematically shows a schematic diagram of attention feature extraction with local attention according to another embodiment of the present disclosure.

[0117] The extraction manner of the attention feature extraction with local attention as shown in FIG. 5B is similar to the extraction manner of the attention feature extraction with global attention as shown in FIG. 5A. The difference lies in that the prompt information of the attention feature extraction with global attention is the position embedding information pos, while the prompt information of the attention feature extraction with local attention is the prompt information pro obtained by combining the position embedding information and the segmentation prompt information.

[0118] As shown in FIG. 5B, the dot product result can be combined with the prompt information, and the combined result is activated by the activation function 550 to obtain the attention weight The value vector 540 is multiplied by the attention weight to obtain the output result of each attention head. Wherein, pro represents the prompt information obtained by combining the position embedding information and the segmentation prompt information.

[0119] According to the embodiments of the present disclosure, through the attention feature extraction with local attention, the pertinence of the channel-enhanced feature map can be improved, the attention degree of the features suitable for object classification is improved by using the segmentation prompt information, thereby realizing image noise reduction, highlighting the image information suitable for the recognition task, and further improving the recognition accuracy.

[0120] The above describes the attention feature extraction manner as shown in FIG. 4 from the attention feature extraction manner with global attention and the attention feature extraction manner with local attention respectively by combining FIG. 5A and FIG. 5B, and the following will explain the combination of the multiple channel-enhanced feature map blocks as shown in FIG. 4.

[0121] According to the embodiments of the present disclosure, the combination of the multiple channel-enhanced feature map blocks to obtain the channel-enhanced feature map can include: sequentially performing channel dimension combination and spatial dimension combination on the multiple channel-enhanced feature map blocks to obtain the channel-enhanced feature map.

[0122] The multiple channel-enhanced feature map blocks are combined in the channel dimension to obtain multiple channel-enhanced sub-feature maps. The multiple channel-enhanced sub-feature maps are combined in the spatial dimension to obtain the channel-enhanced feature map.

[0123] Specifically, each feature map can be split into a plurality of sub-feature maps in the spatial dimension. Each sub-feature map can be split into a plurality of feature map blocks in the channel dimension. The plurality of channel-enhanced feature map blocks corresponding to the plurality of feature map blocks can be combined in the channel dimension to obtain a channel-enhanced sub-feature map. The plurality of channel-enhanced sub-feature maps corresponding to the plurality of sub-feature maps can be combined in the spatial dimension to obtain a channel-enhanced feature map. For example, the plurality of channel-enhanced sub-feature maps can be spliced to obtain the channel-enhanced feature map.

[0124] The feature map combined in the spatial dimension can be directly used as the channel-enhanced feature map. However, it is not limited thereto. The feature map combined in the spatial dimension can also be used as an intermediate channel-enhanced feature map, and the intermediate channel-enhanced feature map can be optimized to obtain the channel-enhanced feature map.

[0125] The feature optimization can include residual connection of the intermediate channel-enhanced feature map and the feature map, but is not limited thereto, and can also include normalization and multilayer perception processing of the intermediate channel-enhanced feature map or the intermediate channel-enhanced feature map connected in residual connection with the feature map.

[0126] According to an embodiment of the present disclosure, the plurality of channel-enhanced feature map blocks are combined in the reverse manner of splitting the feature map, so that the feature map blocks enhanced in the channel feature are combined in positions corresponding to the feature information of the feature map, so that the plurality of local features in the channel-enhanced feature map are combined in association with the semantic features above and below.

[0127] The specific operation of the feature optimization on the feature combined in the spatial dimension is explained above, and the channel dimension combination will be specifically explained below.

[0128] According to an embodiment of the present disclosure, the channel dimension combination of the plurality of channel-enhanced feature map blocks can include: combining the plurality of channel-enhanced feature map blocks in the channel dimension to obtain a channel-enhanced initial sub-feature map. Linear transformation is performed on the channel-enhanced initial sub-feature map to obtain a channel-enhanced sub-feature map.

[0129] The channel dimension combination of the plurality of channel-enhanced feature map blocks to obtain the channel-enhanced initial sub-feature map can include: splicing the plurality of channel-enhanced feature map blocks, for example, using a concat function to splice, to obtain the channel-enhanced initial sub-feature map. However, it is not limited thereto. As long as it is combined according to the splitting relationship of the plurality of feature map blocks corresponding thereto.

[0130] The linear transformation on the channel-enhanced initial sub-feature map can also be understood as linear projection to obtain the channel-enhanced sub-feature map. The linear transformation can be completed by using formula (5) to obtain the channel-enhanced sub-feature map. xa concat(Atten1, Atten2,..., Attena) = Wx + b; formula (5) I )W a +b a ; formula (5)

[0131] wherein x a represents the a-th channel-enhanced sub-feature map, Atten I represents the channel-enhanced feature map block output by the I-th attention head, W a and b a are parameters for linear transformation of the channel-enhanced initial sub-feature map, and concat represents a concatenation function.

[0132] According to the embodiments of the present disclosure, the channel dimension combination and linear transformation of the plurality of channel-enhanced feature map blocks can combine the information of the channel dimension splitting and map it to a mapping space suitable for subsequent processing operations, thereby embodying the integrity and adaptability of the overall recognition process, and further improving the recognition accuracy.

[0133] The operations shown in FIGS. 4, 5A and 5B can be performed using the network structure of the feature extraction module. The network structure of the feature extraction module will be described in detail below in conjunction with FIG. 6.

[0134] FIG. 6 schematically shows a network structure diagram of a feature extraction module according to an embodiment of the present disclosure.

[0135] As shown in FIG. 6, the feature extraction module can include a first normalization layer M610, an attention processing layer M620, a second normalization layer M630 and a multi-layer perceptron processing layer M640. The feature map obtained by vector conversion of the image to be recognized can be taken as an initial feature map 610. The initial feature map is input to the first normalization layer M610 to obtain a feature map. The feature map is sequentially subjected to spatial dimension splitting and channel dimension splitting to obtain a plurality of feature map blocks. The plurality of feature map blocks are input to the attention processing layer M620 to obtain a plurality of channel-enhanced feature map blocks. The plurality of channel-enhanced feature map blocks are respectively subjected to channel dimension combination to obtain a plurality of channel-enhanced sub-feature maps. The plurality of channel-enhanced sub-feature maps are subjected to spatial dimension combination to obtain an intermediate channel-enhanced feature map. The intermediate channel-enhanced feature map and the feature map are connected in residual connection to obtain a gradient-stable feature map. The gradient-stable feature map is sequentially input to the second normalization layer M630 and the multi-layer perceptron processing layer M640 for normalization and multi-layer perceptron processing, and then connected in residual connection with the gradient-stable feature map to obtain a channel-enhanced feature map.

[0136] According to the embodiments of the present disclosure, the features can be optimized by using the normalization processing and the multi-layer perception processing. In addition, the residual connection is performed on the intermediate channel enhanced feature map and the feature map, and the residual connection is performed again on the gradient stable feature map optimized by the normalization and the multi-layer perception, so that the stable propagation of the gradient can be realized through the residual connection. Thus, the information highlighted by the feature enhanced feature map is consistent with the image recognition technology, thereby improving the subsequent classification accuracy.

[0137] Optionally, N stacked feature extraction modules can be set. The input of the nth feature extraction module is the nth feature map, and the output result of the nth feature extraction module is the nth channel enhanced feature map. The output result of the last feature extraction module is the channel enhanced feature map.

[0138] The greater the N setting of the feature extraction module is, the richer the obtained channel enhanced feature map is, but the processing time will be prolonged. N can be reasonably configured according to actual needs, for example, 3-5.

[0139] The operations S220 and S230 shown in FIG. 2 are explained and described in detail above through FIG. 3, FIG. 4, FIG. 5A, FIG. 5B and FIG. 6. The prompt recognition operation in the operation S240 shown in FIG. 2 will be further described below.

[0140] According to the embodiments of the present disclosure, for the operation S240 shown in FIG. 2, the spatial prompt information obtained by performing prompt recognition on the to-be-recognized image can include: determining a plurality of target pixels from the to-be-recognized image based on image information of the to-be-recognized image. Based on the pixel position information of each of the plurality of target pixels, region recognition is performed on the to-be-recognized image to obtain region of interest information. Based on the region of interest information and the pixel position information of each of the plurality of target pixels, the spatial prompt information is obtained.

[0141] The image information can include a plurality of pixel information. Each pixel information can include pixel position information of a pixel, color information of the pixel, brightness information of the pixel, etc.

[0142] Based on the image information of the to-be-recognized image, the plurality of target pixels can be determined from the plurality of pixels in the to-be-recognized image, which can include: screening the plurality of pixels in the to-be-recognized image to obtain the plurality of target pixels. At least one of the pixel position information of the pixel, the color information of the pixel, and the brightness information of the pixel can be used as a screening basis to determine the plurality of target pixels from the plurality of pixels. As long as the target pixel with semantic segmentation prompt function can be screened from the to-be-recognized image.

[0143] The region of interest information can be obtained by performing region identification on the to-be-identified image based on the pixel position information of the target pixels. The region identification can include: marking the target pixels in the to-be-identified image based on the pixel position information of the target pixels. The target pixels that are adjacent in position are grouped together, and at least one target pixel aggregation region is obtained by grouping, which is used as the region of interest. The image information corresponding to the region of interest can be used as the region of interest information.

[0144] The region of interest information and the pixel position information of the target pixels can be used as the spatial prompt information. However, the region of interest information and the pixel position information of at least one target pixel can also be used as the spatial prompt information. The region of interest information and the pixel position information of the target pixels can also be encoded to obtain the spatial prompt information. The encoding manner is not limited, and the encoder can be used for encoding. The attention mechanism can also be used for encoding. As long as the region of interest information and the pixel position information of the target pixels can be combined to obtain a data type that can match the channel-enhanced feature map, the semantic segmentation can be performed.

[0145] The spatial prompt information is obtained based on the region of interest information and the pixel position information of the target pixels. Therefore, the spatial prompt information includes the region local information and the pixel local information of the to-be-identified image, the information granularity of the included image information has a hierarchical nature, and the richness of the prompt information is improved. In addition, the region local information and the pixel local information are determined based on the target pixels related to the target object. Therefore, the richness of the prompt information is improved, the pertinence of the prompt information is improved, and the segmentation accuracy of the semantic segmentation performed on the channel-enhanced feature map based on the spatial prompt information is improved.

[0146] The operation S240 of performing prompt identification on the to-be-identified image to obtain the spatial prompt information is described above with reference to FIG. 2. The operation S240 will be further explained below with reference to FIGS. 7A and 7B.

[0147] FIG. 7A schematically shows a diagram for determining the region of interest information according to an embodiment of the present disclosure.

[0148] As shown in FIG. 7A, the target pixels in the to-be-identified image 710 can be identified to obtain a plurality of target pixels. The target pixels are clustered based on the pixel position information of the target pixels to obtain at least one clustering cluster. As shown in FIG. 7A, three clustering clusters 711 can be obtained. The region of interest 712 can be determined from the to-be-identified image 710 based on the image position information of the at least one clustering cluster. The region of interest information is obtained based on the image information of the region of interest 712.

[0149] The clustering manner can include various manners, and the target pixels with similar or identical color information or brightness information can be clustered and divided into a cluster. However, the clustering manner is not limited to this. As shown in FIG. 7A, the target pixels with similar pixel position information can be clustered and divided into a cluster. For example, the pixel distance between the target pixels can be determined based on the pixel position information of the target pixels. The adjacent target pixels with a pixel distance less than or equal to a pixel distance threshold can be divided into a cluster.

[0150] The image position information of the cluster can include, but is not limited to, the image position information of the smallest box for enclosing all the target pixels in the cluster. For example, the image position information of the four corners of the box, but is not limited to this, and for example, the image position information of one corner of the box and the height and width of the box.

[0151] The region of interest information can be obtained based on the image position information of the clusters. However, the region of interest information is not limited to this. At least one target cluster can be determined from the clusters, and the image position information of the at least one target cluster can be used as the region of interest information. The clusters with an area greater than an area threshold can be selected as the target clusters from the clusters according to the area of the box. The clusters can be sorted in descending order of the area of the box, and the first L clusters can be selected as the target clusters from the clusters. In the case of less than or equal to L clusters, the L clusters can be directly used as the target clusters. For example, the image position information of the i th box S i can be represented as (x i, y i, h i, w i ), (x i, y i ) represents the pixel position information of the corner b i of the i th box S i, h i and w i represent the height and width of the corner b i of the i th box S i, respectively. The region of interest information can include the image position information of the q boxes, represented as |S 1, S 2, …, S q |. bi bi bi bi bi bi bi bi bi bi bi b1 bq

[0152] The region of interest information can be determined from the image to be recognized in the above manner, the target pixels obtained through the preliminary screening of the image information can be further screened by the clustering manner, the target pixels not in the cluster or with low attention in the cluster can be filtered out, the spatial cue information is denoised through the two screening manners, and the prompting ability of the spatial cue information for assisting semantic segmentation is improved.​​​​​​​​​​​​​

[0153] FIG. 7B schematically shows a diagram for determining target pixels according to an embodiment of the present disclosure.

[0154] As shown in FIG. 7B, the image information of the image to be recognized 710 can be subjected to a binarization process to obtain a binarized image 720. Based on the binarized image 720, a plurality of target pixels as shown in FIG. 7A can be obtained.

[0155] The image information can include pixel information of each of the plurality of pixels. The pixel information can include one or more of a pixel grayscale value, a pixel color, or a pixel brightness.

[0156] The pixel information of the pixel and a pixel binarization threshold value can be compared. The pixels with pixel information less than or equal to the pixel binarization threshold value can be masked in the image to be recognized, and the pixels with pixel information greater than the pixel binarization threshold value can be marked in the image to be recognized to obtain a binarized image. Illustratively, the marking manner can include converting the target pixel information using a predetermined pixel information. The pixel binarization threshold value can be pre-set. The pixel binarization threshold value can be adjusted according to actual conditions and related information of the target object to be subjected to semantic segmentation, so as to adapt to subsequent semantic segmentation operations.

[0157] For example, the pixel information of the pixel in the image to be recognized can be represented as v(x, y) f , the pixel binarization threshold value can be represented as threshold, and the pixel information of the pixel in the binarized image can be represented as v(x, y) b . In the case of v(x, y) f > threshold, v(x, y) b = 1. In the case of v(x, y) f ≤ threshold, v(x, y) b = 0.

[0158] The plurality of pixels marked in the binarized image can be used as target pixels. From the binarized image, the pixel position information of each target pixel can be determined.

[0159] By subjecting the image to be recognized to a binarization process, the pixels in the image to be recognized can be screened, and the subsequent processing amount can be reduced, thereby improving the accuracy of determining the region of interest information and speeding up the efficiency of obtaining the spatial prompt information.

[0160] The above describes in detail how to determine the target pixels and the region of interest in combination with FIG. 7A and FIG. 7B. How to obtain the spatial prompt information based on the pixel position information of the target pixels and the region of interest information will be described in detail below.

[0161] Exemplarily, the multiple target pixels in the region of interest can be determined based on the pixel position information of the target pixels. The pixel position information of the multiple target pixels and the region of interest information of all or part of the multiple target pixels in the region of interest can be taken as the spatial cue information. However, this is not limited thereto. A predetermined number of target pixels can be selected from the multiple target pixels according to a predetermined interval, and the pixel position information of the predetermined number of target pixels and the region of interest information can be taken as the spatial cue information.

[0162] The pixel-level local position information and the region-level local position information are taken together as the spatial cue information, thereby improving the richness and hierarchy of the spatial cue information. In addition, the binary processing is performed on the multiple pixels in the image to be recognized, the target pixels related to the semantic segmentation target object are recognized, and the region of interest information and the spatial cue information are determined based on the position information of the target pixels, thereby improving the prompting effect of the spatial cue information.

[0163] The operations S220 and S230 shown in FIG. 2 are described in detail above by combining FIG. 3, FIG. 4, FIG. 5A, FIG. 5B and FIG. 6, and the operation S240 shown in FIG. 2 is described in detail above by combining FIG. 7A and FIG. 7B. The operation S250 shown in FIG. 2, the semantic segmentation of the channel-enhanced feature map based on the spatial cue information, will be explained below.

[0164] According to an embodiment of the present disclosure, for the operation S250 shown in FIG. 2, the semantic segmentation of the channel-enhanced feature map based on the spatial cue information, the recognition map for the target object is obtained.

[0165] The processing mode of the semantic segmentation can include at least one of attention feature extraction, semantic segmentation feature adaptation, semantic segmentation feature optimization, encoding, and decoding.

[0166] The semantic segmentation of the channel-enhanced feature map is to perform feature enhancement on the features matched with the target object, thereby obtaining the recognition map with segmentation function marked with the same pixel category.

[0167] In an example, the feature optimization can be performed on the channel-enhanced feature map to obtain the feature-enhanced feature map.

[0168] For example, the feature optimization can be performed on the channel-enhanced feature map at least once to obtain the feature-enhanced feature map. Each round of feature optimization can include convolution and normalization.

[0169] The convolution can include a 3*3 convolution kernel to reduce the vector dimension of the channel-enhanced feature map by convolution processing. The normalized processing is performed on the feature map after the convolution. Thus, the feature map with task feature enhancement is a 256-dimensional reduced feature map, for example, reduced by 16 times.

[0170] The above specifically explains a specific operation of performing semantic segmentation on a channel-enhanced feature map. The network structure of a semantic segmentation module capable of performing the semantic segmentation operation will be specifically explained below in combination with FIG. 8A.

[0171] FIG. 8A schematically shows a network structure diagram of a semantic segmentation module according to a related example.

[0172] As shown in FIG. 8A, the channel-enhanced feature map output by the feature extraction module M810 is input into the semantic segmentation module M820 to obtain the recognition map 810 for the target object.

[0173] As shown in FIG. 8A, the semantic segmentation module M820 includes N1 stacked feature enhancement modules, each of which includes a convolution layer M821 and a pooling layer M822.

[0174] The semantic segmentation operation as shown in FIG. 2 can be implemented by the convolution and pooling specific operations as shown in FIG. 8A. However, it is not limited thereto. The semantic segmentation operation as shown in FIG. 2 can also be implemented by specific operations such as attention feature extraction and task feature optimization. Different from the semantic segmentation processing mode as shown in FIG. 8A will be explained below.

[0175] In another example, based on the target prompt information, the channel-enhanced feature map is subjected to task feature enhancement to obtain a recognition map for the target object, which can include: based on the target prompt information, the channel-enhanced feature map is subjected to attention feature extraction to obtain an attention feature map. The attention feature map is subjected to semantic segmentation feature optimization to obtain the recognition map.

[0176] The target prompt information can include the spatial prompt information, the position embedding information and the segmentation prompt information described above. The position embedding information indicates that the feature map block corresponds to an image position in the image to be recognized. The segmentation prompt information indicates the object type of the target object expected to be segmented. The spatial prompt information indicates the pixel position information and the region of interest information of the target object in the image to be recognized.

[0177] The manner of performing attention feature extraction on the channel-enhanced feature map is similar or identical to the manner of performing attention feature extraction on a plurality of feature map blocks, which will not be described herein again.

[0178] The semantic segmentation feature optimization can include task adaptation of the attention feature map. For example, after processing the channel enhanced feature map by the semantic segmentation module, a recognition map is obtained. The recognition map is an image that segments the pixel categories in the image to be recognized. The recognition map can also be classified to obtain a recognition result for the target object. Therefore, after the semantic segmentation feature optimization processing, the adaptability of the recognition map for subsequent classification operations can be improved. The main purpose of the semantic segmentation feature optimization includes improving the adaptability of the recognition map to the classification task.

[0179] Specifically, the semantic segmentation feature optimization can include, but is not limited to, at least one of the following: feature extraction, feature mapping, feature adaptation, normalization, etc. of the attention feature map.

[0180] The attention feature map can be directly used as a recognition map. However, compared with the semantic segmentation method that only performs attention feature extraction, the combination of attention feature extraction and semantic segmentation feature optimization for semantic segmentation can improve the effect of semantic segmentation, so that the processing stability of the recognition map is adapted to the hardware device, and at the same time, the recognition map is adapted to the classification task.

[0181] The attention feature extraction can be performed on the channel enhanced feature map based on the target prompt information, thereby improving the attention weight of the task feature of the channel enhanced feature map for performing target object category recognition, and improving the importance of the features related to the target object category. In addition, the semantic segmentation feature optimization is performed on the attention feature map, which further plays a role of task adaptation of the classification of the recognition map, so that the recognition result of the target object category based on the recognition map has high accuracy.

[0182] The following will describe a semantic segmentation module different from that shown in FIG. 8A in combination with FIG. 8B.

[0183] FIG. 8B schematically shows a network structure diagram of a semantic segmentation module according to another embodiment of the present disclosure.

[0184] As shown in FIG. 8B, the semantic segmentation module M820’ can include a cross-attention feature extraction layer M821’, an adapter M822’, and an optimizer M823’. The channel enhanced feature map and the target prompt information can be input into the cross-attention feature extraction layer M821’ to obtain an initial attention feature map. The initial attention feature map is input into the adapter M822’ to obtain an attention feature map. The attention feature map is input into the optimizer M823’ to obtain a recognition map 810.

[0185] Optionally, N' semantic segmentation modules can be set up in a stack. The input data of the nth' semantic segmentation module is taken as the feature map enhanced in the nth' channel, and the output data of the nth' semantic segmentation module is taken as the feature map enhanced in the nth' task feature. The output data of the last semantic segmentation module is taken as the recognition map.

[0186] The greater the N' of the semantic segmentation module is set, the richer the recognition map obtained will be, but the processing time will be prolonged. N' can be reasonably configured according to actual needs.

[0187] The operation performed by the cross-attention feature extraction layer shown in FIG. 8B will be described below.

[0188] According to an embodiment of the present disclosure, based on the target prompt information, the channel-enhanced feature map is subjected to attention feature extraction to obtain an attention feature map, which can include: based on the target prompt information, the channel-enhanced feature map is subjected to cross-attention feature extraction to obtain an initial attention feature map. The initial attention feature map is subjected to semantic segmentation feature adaptation to obtain the attention feature map.

[0189] The cross-attention feature extraction can include: feature extraction or feature encoding in the manner of cross-attention mechanism. Only the manner of cross-attention mechanism can be used, or the manner of combining cross-attention mechanism and other operations such as convolution, pooling, normalization, and multi-layer perception can be used, as long as the feature extraction manner can improve the features related to the segmentation target object. Exemplarily, the decoder in the Transformer (encoder-decoder) configured with cross-attention can be used for cross-attention feature extraction.

[0190] The channel-enhanced feature map can be subjected to attention feature extraction by using the cross-attention mechanism to obtain the attention feature map. The channel-enhanced feature map can be subjected to at least one of spatial dimension splitting and channel dimension splitting, and then subjected to attention feature extraction. As long as the attention feature map can be obtained.

[0191] Optionally, the channel-enhanced feature map can be subjected to spatial dimension splitting, the target prompt information is taken as the prompt information, and the channel-enhanced feature map after spatial dimension splitting is subjected to cross-attention feature extraction to obtain an initial attention feature map.

[0192] The channel-enhanced feature map after spatial dimension splitting is subjected to cross-attention feature extraction to obtain an initial attention feature map, which can include: performing self-attention feature extraction on the feature map after spatial dimension splitting to obtain an image embedding. The self-attention feature extraction can be completed through 8 self-attention heads. In the self-attention feature extraction, the channel dimension can be set to 256. Cross-attention feature extraction is performed from the target prompt information to the image embedding, wherein the target prompt information serves as a query vector and the image embedding serves as a key vector and a value vector. The target prompt information is subjected to multi-layer perceptron processing with a hidden dimension of 2048 and activation processing to obtain updated target prompt information. Cross-attention feature extraction is performed from the image embedding to the updated prompt information. The dimension of the cross-attention feature extraction can be set to 128. After each self-attention feature extraction and cross-attention feature extraction, a residual connection and a normalization layer are added. Before performing each cross-attention feature extraction, the original target prompt information is added back to the query vector to better perform task prompting. Multiple cross-attention feature extraction layers can be set. The target prompt information and the image embedding output by the previous cross-attention feature extraction layer can be used as input data for the next cross-attention feature extraction layer.

[0193] According to an embodiment of the present disclosure, the channel-enhanced feature map is subjected to attention feature extraction in the manner of cross-attention feature extraction, which can enhance the task features of the channel-enhanced feature map based on the target prompt information, thereby highlighting the features related to the object category in the initial attention feature map.

[0194] The specific operation of the cross-attention feature extraction layer shown in FIG. 8B is explained above, and the adapter shown in FIG. 8B for performing semantic segmentation feature adaptation will be specifically explained below in conjunction with FIG. 9.

[0195] The adapter can be set to perform semantic segmentation feature adaptation on the initial attention feature map to obtain an attention feature map.

[0196] The semantic segmentation feature adaptation can include: a manner of processing the initial attention feature map by the adapter that has learned semantic segmentation-related knowledge. It can be understood that, for the network parameters of the pre-trained model that have general knowledge, network parameter adjustment is performed on the pre-trained model in the semantic segmentation task, which is time-consuming and laborious. The parameter-light adapter can be configured in the pre-trained model, and the network parameters of the pre-trained model are frozen in the fine-tuning process, and only the adapter is learned about the semantic segmentation task-related knowledge, and the network parameters of the adapter are adjusted, thereby reducing the network parameters in the fine-tuning process and improving the training speed.

[0197] FIG. 9 schematically shows a network structure diagram of an adapter according to an embodiment of the present disclosure.

[0198] As shown in FIG. 9, the adapter can be a bottleneck structure. The adapter M910 can include a down liner layer M911, a GELU (Gaussian Error Linear Unit) layer M912, and an up liner layer M913, also referred to as an upper projection layer.

[0199] As shown in FIG. 9, the adapter can further include a residual connection. The pre-adaptation feature and the post-adaptation feature are connected in residual.

[0200] The initial attention feature map can be down-sampled by the down liner layer to obtain a first adaptation feature map. Specifically, the initial attention feature map x can be down-sampled by the down liner layer with parameters W down ∈R d*m to project the original d-dimensional feature to a small m-dimensional feature. The down liner layer parameters further include b down .

[0201] The first adaptation feature map can be non-linearly transformed by the GELU layer to obtain a second adaptation feature map. The GELU layer can use a non-linear activation function GELU, thereby providing a smoother gradient.

[0202] The second adaptation feature map can be up-sampled by the up liner layer to obtain a third adaptation feature map. The third adaptation feature map can be obtained by up-sampling the second adaptation feature map with parameters W up ∈R m*d to restore the feature to a d-dimensional space. The up liner layer parameters further include b up .

[0203] The attention feature map and the third adaptation feature map are connected in residual to obtain an adaptation feature map.

[0204] The intermediate dimension m is limited by the dimension d, m << d. The initial attention feature map x is adapted to obtain an adaptation feature map x' by feature adaptation calculation, which can be referred to as formula (6). x' = W up ·GELU(W down x + b down )+b up +x; formula (6)

[0205] According to an embodiment of the present disclosure, the initial attention feature map is processed by feature adaptation, which can alleviate the overfitting problem and further improve the accuracy of the recognition result obtained based on the recognition graph. In addition, by using the adapter, the optimization parameters in the training process can be reduced, and the training efficiency of the image recognition model can be improved.

[0206] The adapter as shown in FIG. 8B is specifically described above, and the optimizer as shown in FIG. 8B will be specifically explained below in combination with FIG. 10.

[0207] According to an embodiment of the present disclosure, the semantic segmentation feature optimization is performed on the attention feature map to obtain a recognition map, which can include: performing multi-layer perceptron processing and semantic segmentation feature adaptation on the attention feature map respectively to obtain a first intermediate feature map and a second intermediate feature map. The attention feature map, the first intermediate feature map and the second intermediate feature map are connected by a residual connection to obtain a feature map with enhanced task features.

[0208] The semantic segmentation feature adaptation manner of the attention feature map is the same as the feature adaptation manner of the initial attention feature map, and the difference is that a shrinkage factor s is set to adapt the second intermediate feature map to the first intermediate feature map and the attention feature map. The input data of the given multi-layer perceptron processing layer MLP (Multi-Layer Perceptron) is the attention feature map x m The feature map after the normalization processing, for example, LN(x m ) and its recognition map are x′ m , and the calculation formula performed by the feature optimization of the optimizer can be seen from formula (7). x′ m = s·(W up ·GELU(W down LN(x m )+b down )+b up )+MLP(LN(x m ))+x m ; formula (7)

[0209] FIG. 10 schematically shows a network structure schematic diagram of an optimizer according to an embodiment of the present disclosure.

[0210] As shown in FIG. 10, the intermediate feature map for inputting into the cross-attention feature extraction layer M1020 can be inputted into the first normalization layer M1010 to obtain a normalized intermediate feature map. The normalized intermediate feature map is subjected to spatial dimension splitting to obtain a plurality of intermediate sub-feature maps. The plurality of intermediate sub-feature maps are sequentially processed by the cross-attention feature extraction layer M1020 and the first adapter M1030 to obtain a plurality of attention intermediate sub-feature maps. The plurality of attention intermediate sub-feature maps are subjected to spatial dimension combination to obtain an attention feature map. The attention feature map is inputted into the second normalization layer M1040 in the optimizer to obtain a normalized feature map. The normalized feature map is inputted into the multi-layer perception processing layer M1050 and the second adapter M1060 to obtain a first intermediate feature map and a second intermediate feature map, respectively. The attention feature map, the first intermediate feature map, and the second intermediate feature map subjected to the contraction factor processing are subjected to residual connection to obtain a task feature enhanced feature map.

[0211] The channel enhanced feature map outputted by the feature extraction module as shown in FIG. 6 can be taken as the intermediate feature map as shown in FIG. 10. However, it is not limited thereto. For example, the channel enhanced feature map can be processed by the semantic segmentation module stack as shown in FIG. 8B.

[0212] According to an embodiment of the present disclosure, the multi-layer perception processing and feature adaptation are operated in parallel, which can optimize the attention feature map from different aspects, and further improve the processing accuracy and stability.

[0213] An image recognition model can be designed, which is called by the processor as shown in FIG. 1A or as shown in FIG. 1B. The image recognition model is used to recognize the to-be-recognized image to obtain a recognized image. The specific network structure of the image recognition model comprising the feature extraction module as shown in FIG. 6 and the semantic segmentation module as shown in FIG. 8B will be explained below in combination with FIG. 11A and FIG. 12B.

[0214] FIG. 11A schematically shows a schematic diagram of an image recognition model according to an embodiment of the present disclosure.

[0215] As shown in FIG. 11A, the image recognition model can comprise an image embedding module M1110, a feature extraction module M1120, a task prompt embedding module M1130, a semantic segmentation module M1140, and a classification module M1150.

[0216] As shown in FIG. 11A, an image to be recognized 1110 can be input into an image embedding module M1110 to obtain a feature map 1120. The feature map 1120 and position embedding information pos are input into a feature extraction module M1120 to obtain a channel-enhanced feature map. The position embedding information pos, segmentation prompt information, and spatial prompt information are input into a task prompt embedding module M1130 to obtain target prompt information. The channel-enhanced feature map and the target prompt information are input into a semantic segmentation module M1140 to obtain a recognition image 1130. The recognition image 1130 is input into a classification module M1150 to obtain a recognition result.

[0217] Optionally, the image embedding module can include a convolutional layer with a step size of 16 and a convolution kernel of 16*16, and the image to be recognized can be convolved into a small-size high-dimensional feature map.

[0218] Optionally, the feature extraction module, which can also be referred to as a mask encoder, can include N stacks, for example, N is 12. The unit channel dimension of the feature block is 768, and the self-attention head can include 12.

[0219] Optionally, the input data of the task prompt embedding module, for example, the segmentation prompt information, can include spatial prompt information and text as segmentation prompt information obtained from target pixel information and region of interest information, etc. In this example, the segmentation prompt information can refer to the object category of the target object, for example, the object category of the crack as the target object, but is not limited thereto, and the input data can also be simplified to “none”, which is a default learnable segmentation prompt information. It is added to each position of the channel-enhanced feature map.

[0220] Optionally, the semantic segmentation module, which can also be referred to as a mask decoder or a bidirectional transformer, can interact the channel-enhanced feature map with the segmentation prompt information through cross-attention feature extraction, so that the features adapted to the task are highlighted, thereby improving the recognition accuracy of the recognition result based on the recognition image.

[0221] FIG. 11B schematically shows a schematic diagram of an image recognition model according to another embodiment of the present disclosure.

[0222] The difference between the image recognition model shown in FIG. 11A and the image recognition model shown in FIG. 11B is that the image recognition model shown in FIG. 11B can include a first image embedding module M1111 and a second image embedding module M1112.

[0223] As shown in FIG. 11B, the to-be-recognized image 1110 can be input into the first image embedding module M1111 to obtain a feature map 1120. The feature map 1120 and the position embedding information are input into the feature extraction module M1120 to obtain a channel-enhanced feature map. The channel-enhanced feature map is input into the second image embedding module M1112 to obtain a channel-enhanced embedding map. The position embedding information pos, the segmentation prompt information, and the spatial prompt information are input into the task prompt embedding module M1130 to obtain target prompt information. The channel-enhanced embedding map and the segmentation prompt information are input into the semantic segmentation module M1140 to obtain a recognition image 1130. The recognition image 1130 is input into the classification module M1150 to obtain a recognition result.

[0224] Optionally, the second image embedding module can include a transpose convolution layer, a normalization layer, and an activation layer. Two layers of transpose convolution layers with a stride of 2 and a kernel size of 2*2 can be used to upsample the channel-enhanced feature map of 1 / 16 size to 1 / 4 size. The channels of the transpose convolution layers are 64 and 32, respectively. After convolution, the input is activated by the activation layer, and a normalization layer is added between the transpose convolution layer and the activation layer.

[0225] Optionally, the feature extraction module as shown in FIG. 11A or 11B can include the network structure of the feature extraction module as shown in FIG. 6. However, it is not limited thereto. It can also include the network structure of the semantic segmentation module as shown in FIG. 8B on the basis of the network structure of the feature extraction module as shown in FIG. 6. Specifically, the overall network structure of the feature extraction module and the semantic segmentation module as shown in FIG. 8B can be used as the network structure of the feature extraction module as shown in FIG. 11A or 11B.

[0226] The second image embedding module is arranged in the image recognition model, which can adapt the vector dimensions of the output result of the feature extraction module and the input result of the semantic segmentation module, so as to improve the overall processing efficiency and overall stability of the image recognition model.

[0227] The recognition image obtained by processing the to-be-recognized image by using the image recognition model as shown in FIG. 11A or 11B can be understood as an object segmentation image with a masked background. The background masking effect of the recognition image processed by the image recognition model as shown in FIG. 11A or 11B will be highlighted below in combination with FIG. 12.

[0228] FIG. 12 schematically shows a schematic diagram of a to-be-recognized image and a task feature-enhanced feature map according to an embodiment of the present disclosure.

[0229] As shown in FIG. 12, the first column is eight to-be-recognized images containing different crack objects. After processing by the above image recognition model, the corresponding task feature-enhanced feature maps are obtained, as shown in the second column.

[0230] As shown in FIG. 12, after the plurality of to-be-recognized images obtained for different shooting objects in different shooting environments are processed by the image recognition model, a mask segmentation image about the crack can be obtained.

[0231] Therefore, the image features in the to-be-recognized image that are irrelevant to the crack segmentation task can be masked by using the image recognition model, and the crack features related to the crack segmentation task can be highlighted, so as to obtain the mask segmentation image about the crack. The recognition result obtained based on the mask segmentation image about the crack is accurate and effective.

[0232] FIG. 12 schematically shows a plurality of task feature enhanced feature maps of different types, and the classification of the task feature enhanced feature map will be described in detail below.

[0233] According to an embodiment of the present disclosure, based on the task feature enhanced feature map corresponding to the to-be-recognized image, obtaining the recognition result of the object in the to-be-recognized image can include: classifying the object in the to-be-recognized image based on the task feature enhanced feature map to obtain the recognition result.

[0234] The task feature enhanced feature map can be processed by using a classification layer to obtain the recognition result of the object category representing the object. The network structure of the classification layer is not limited, for example, can include a multilayer perceptron and an activation function.

[0235] According to an embodiment of the present disclosure, the object in the to-be-recognized image is classified by using the task feature enhanced feature map. The features in the task feature enhanced feature map that match the classification task are focused on, and the background information irrelevant to the classification task is masked, so that the recognition accuracy of the object category can be improved, and the influence of noise or low resolution can be reduced.

[0236] The image recognition model shown in FIG. 11A or FIG. 11B can be trained, so that the trained image recognition model can accurately and effectively recognize the abnormal object. The training method of the image recognition model related above will be explained in combination with FIG. 13.

[0237] FIG. 13 schematically shows a flowchart of a training method of an image recognition model according to an embodiment of the present disclosure.

[0238] As shown in FIG. 13, the training method of the image recognition model includes operations S1310 to S1360.

[0239] In operation S1310, the sample feature map of the sample input image is split to obtain a plurality of sample feature map blocks.

[0240] In operation S1320, the plurality of sample feature map blocks are subjected to attention feature extraction to obtain a channel enhanced sample feature map.

[0241] In operation S1330, prompt recognition is performed on the sample input image to obtain sample space prompt information.

[0242] In operation S1340, based on the sample space prompt information, semantic segmentation is performed on the sample channel-enhanced feature map to obtain a sample recognition map for the target object.

[0243] In operation S1350, the sample recognition map is classified to obtain a sample recognition result.

[0244] In operation S1360, based on the sample recognition result and a sample label corresponding to the sample input image, the image recognition model is trained.

[0245] The sample label indicates a category result of the target object in the sample input image.

[0246] Optionally, the sample input image, the sample feature map block, the sample feature map, the sample channel-enhanced feature map, the sample recognition map, and the sample recognition result shown in FIG. 13 are only different in terminology from the to-be-recognized image, the feature map block, the feature map, the channel-enhanced feature map, the recognition map, and the recognition result shown in FIG. 8A or FIG. 8B, and there is no difference in meaning and operation performed. Details are not repeated here.

[0247] Exemplarily, the image recognition model shown in FIG. 11B can be used to perform operations S1310-S1360 shown in FIG. 13 on the sample input image. For example, the sample input image is input into the first image embedding module of the image recognition model to obtain a sample feature map. The sample feature map is subjected to spatial dimension splitting to obtain a sample sub-feature map. The sample sub-feature map is subjected to channel dimension splitting to obtain a plurality of sample feature map blocks. The sample feature map blocks and learnable position embedding information are input into the feature extraction module to perform attention feature extraction to obtain a plurality of sample channel-enhanced feature maps. The plurality of sample channel-enhanced feature maps are input into the second image embedding module to obtain a sample channel-enhanced embedding map. The sample channel-enhanced embedding map and learnable sample space prompt information are input into the semantic segmentation module to obtain a sample recognition map. The sample recognition map is input into the classification module to obtain a sample recognition result.

[0248] According to embodiments of the present disclosure, the above-described manner can be used to perform channel dimension fine-grained feature extraction on the sample input image, and then use the sample space prompt information to extract a recognition map matching the task, so that the sample recognition map is rich in features suitable for the object category used to recognize the target object, thereby obtaining a sample recognition result with high precision based on the sample recognition map. In order to train the image recognition model by using the sample recognition result with high precision and the sample label, the training efficiency is high while the training precision is high.

[0249] The above describes how to train the image recognition model as a whole through FIG. 13. The following will describe in detail how to obtain sample image pairs to improve the training effect.

[0250] The sample image pair can include a sample input image and a sample label.

[0251] In an example, an open-source image dataset can be used as a sample image pair for training. The open-source image dataset can include CRACK500, GAPS384, CFD, CRACKTREE200, and DEEPCRACK.

[0252] Compared with using a single open-source image dataset, using open-source image datasets from various sources can enrich the data so that the image recognition model can perceive objects of different scales, such as cracks, thereby improving the generalization ability of the image recognition model.

[0253] Optionally, the open-source image dataset can be preprocessed to unify the resolution to 448*448.

[0254] In another example, the sample input image in the open-source image dataset can also be subjected to data augmentation processing to obtain a sample image pair. For example, the data augmentation techniques can include, but are not limited to, random rotation and random flipping, such as random horizontal flipping and random vertical flipping, and can also include random cropping, image translation, weak color distortion, image distortion, and the like.

[0255] Compared with random cropping, image translation, and image distortion, random rotation or random flipping can avoid damaging the characteristics of the object and improve the image recognition model to quickly and accurately learn the features for highlighting the object category during the training process.

[0256] The open-source image dataset usually has a long-tail distribution, for example, most sample image pairs are labeled as sample labels of normal objects, because sample image pairs labeled as sample labels of abnormal objects are rare. These unbalanced open-source image datasets can cause the trained image recognition model to overfit, that is, most abnormal objects are classified as normal objects. Therefore, relying only on random horizontal flipping and random vertical flipping to increase the sample size of the open-source image dataset is not sufficient to alleviate the overfitting problem.

[0257] Therefore, data augmentation can be further performed in the following manner, so that the image recognition model can effectively learn the object feature knowledge while avoiding overfitting due to the rich training sample size.

[0258] According to an embodiment of the present disclosure, before operation S1310 as shown in FIG. 13 is performed, the training method of the image recognition model can further include: generating a sample image pair.

[0259] In an example, generating the sample image pair can include: fusing a target region image of the first sample image and a region image of the second sample image except the target region to obtain a sample input image. A sample label corresponding to the sample input image is obtained based on a label of the first sample image, a label of the second sample image, and a fusion ratio. The fusion ratio includes a proportion of the target region in the first sample image region.

[0260] The first sample image x A may include a cropped region and a target region that is not cropped. The cropped region can include a bounding box coordinate B(rxs, rys, rxe, rye), where (rxs, rys) represents a top-left corner coordinate point and (rxe, rye) represents a bottom-right corner coordinate point. w = rxe-rxs is the width of the cropped region, h = rye-rys is the height of the cropped region, W and H are the width and height of the first sample image respectively. The proportion of the target region in the first sample image, i.e., the fusion ratio, can be represented as λ = 1-(w*h) / (W*H).

[0261] Fusing the target region of the first sample image and the region image of the second sample image except the target region can include: image splicing the target region image of the first sample image that is not cropped and the region image of the second sample image corresponding to the cropped region to obtain the sample input image. For example, the sample input image can be calculated by formula (8).

[0262] wherein M represents the first sample image x A is a binary mask of the cropped region, ⊙ represents element-wise multiplication, x B is the second sample image. (1-M) represents the second sample image x B is a binary mask of the target region, is the sample input image.

[0263] The sample label of the sample input image can be obtained by using a weighted summation method, as shown in formula (9).

[0264] wherein λ represents the fusion ratio. y A is the sample label of the first sample image. y B is the sample label of the second sample image. is the sample label of the sample input image.

[0265] According to an embodiment of the present disclosure, the sample image pair is obtained in the above manner, which not only realizes zero manual sample labeling, but also reduces the damage to the object features while improving the diversity of the sample image pair and avoiding the imbalance of the training samples. The image recognition model learns effective feature knowledge while avoiding overfitting, thereby improving the stability and robustness of the object recognition of the trained image recognition model.

[0266] The above embodiment is to cut a part of the first sample image and a part of the second sample image respectively, and then splice the two parts of images to obtain the sample input image. The difference between the following embodiment and the above embodiment is that the second sample image is shrunk, and the shrunk second sample image and part of the first sample image are spliced to obtain the sample input image. In this way, the complete image information of the second sample image is retained.

[0267] Specifically, the target region image of the first sample image and the shrunk second sample image are fused to obtain the sample input image. Based on the label of the first sample image, the label of the second sample image, the fusion ratio and the shrinkage ratio, the sample label of the sample input image is obtained. The fusion ratio includes the proportion of the target region to the region of the first sample image, and the shrinkage ratio is determined based on the target region and the region of the second sample image.

[0268] The shrunk second sample image can include: the second sample image shrunk to the size of the cropped region.

[0269] Fusing the target region of the first sample image and the shrunk second sample image can include: splicing the target region of the first sample image that is not cropped and the second sample image shrunk to the size of the cropped region to obtain the sample input image.

[0270] The first sample image x A may include a cropped region and a target region that is not cropped. The cropped region can include a bounding box coordinate B(rxs, rys, rxe, rye), where (rxs, rys) represents the upper left corner coordinate point and (rxe, rye) represents the lower right corner coordinate point. w=rxe-rxs is the width of the cropped region, h=rye-rys is the height of the cropped region, W and H are the width and height of the first sample image respectively. The proportion of the target region to the first sample image, i.e. the fusion ratio, can be represented as λ=1-(w×h) / (W×H).

[0271] The second sample image x B has a width and a height of W×H. The second sample image x B is shrunk to a shrunk second sample image x′ B , which has a width and a height of W′×H′. The shrinkage ratio of the second sample image is Bis (W' x H') / (W x H). x' B = x B ·∈ B The shrinkage ratio of the first sample image is ∈ A = 1. The fusion ratio λ A = 1 - (w x h) / (W x H). The fusion ratio λ B may include W' x H' is equal to w x h.

[0272] For example, the sample input image can be calculated by formula (10).

[0273] wherein M represents a binary mask of the target region, and represents the first sample image x A The binary mask of the cropped region, and represents the first sample image x B represents the shrunk second sample image. (1 - M) represents the shrunk second sample image x' B The binary mask of the target region, represents the sample input image.

[0274] The sample label of the sample input image can be obtained by formula (11).

[0275] wherein y A represents the sample label of the first sample image. y B represents the sample label of the second sample image. represents the sample label of the sample input image.

[0276] According to an embodiment of the present disclosure, pasting the shrunk complete image of the second sample image onto the cropped region of the first sample image can enable the generated sample input image to contain the complete image information of the second sample image. In this way, the entire image information including the object and the object context is retained.

[0277] The above describes the specific operation of obtaining the sample image pair before operation S1310 shown in FIG. 13, to improve the training effect by using the sample image pair. The following describes operation S1360 shown in FIG. 13, how to train the image recognition model, to improve the training effect by using the loss function.

[0278] In an example, based on the sample recognition result and the sample label corresponding to the sample input image, training the image recognition model shown in operation S1350 in FIG. 13 can include: the sample recognition result and the sample label can be input into the loss function to obtain a loss value. Based on the loss value, the parameters in the image recognition model are adjusted to obtain the trained image recognition model.

[0279] The loss function can include a classification loss function, but is not limited thereto, and can also include a mask loss function. Illustratively, the classification loss function can include a cross-entropy loss function. The mask loss function can include a pixel loss function, but is not limited thereto. The loss function can also include a combination of the classification loss function and the pixel loss function. See Equations (12)-(14). L = βL CE + (1 - β)L Dice ; Equation (14)

[0280] where y represents a sample label, and takes a value of 0 or 1, and is used to represent the probability of the presence or absence of a target object in a mask region. represents a sample recognition result. X represents a sample task feature enhanced feature map, and Y represents an object mask image in a sample input image. The parameter β is a weighting coefficient, which can be set to 0.2.

[0281] In an example, the adjustment of the learning rate can include: in a predetermined number of iterations, the learning rate is linearly increased from zero to an initial learning rate of 0.0004. However, it is not limited thereto. As long as it is a suitable adjustment method.

[0282] In a related example, the classification loss function can include a cross-entropy loss function. The input of the cross-entropy loss function includes the sample recognition result and the sample label. However, considering that in the case of an object being a crack, due to the imbalance of the size of the crack, the cross-entropy loss value tends to converge rapidly, for example, the cross-entropy loss value is zero, so that the background dominates in the cross-entropy loss. Further, the recognition accuracy of the trained image recognition model is low.

[0283] In the embodiments of the present disclosure, the recognition result of the object class and the mask image of the object are used as the basis for adjusting the parameters at the same time, which can improve the balance of convergence, and further improve the efficiency of parameter optimization.

[0284] The above describes how to train the loss function of the image recognition model as shown in operation S1360 of FIG. 13. The use of simplified model parameters to improve the training effect will be described below in conjunction with FIG. 14.

[0285] In the embodiments of the present disclosure, for operation S1360 as shown in FIG. 13, training the image recognition model based on the sample segmentation image corresponding to the sample feature map sequence and the sample label corresponding to the sample input image can include: performing matrix decomposition on the model parameter matrix of the image recognition model to obtain a low-order model parameter matrix. The parameters in the low-order model parameter matrix are adjusted using the sample recognition image and the sample label.

[0286] The low-order model parameter matrix can also be referred to as a low-order adaptive model parameter matrix. The model parameter matrix can be decomposed by using a parameter optimization method.

[0287] For example, the original model parameter matrix in the image recognition model is a large full-rank matrix. The original model parameter matrix in the image recognition model can be mapped to a subspace with a smaller parameter quantity by matrix decomposition to obtain a low-order model parameter matrix.

[0288] For example, the pre-training model parameter matrix W0 e R d×k and an additional update parameter matrix AW e R d×k as the model parameter matrix in the image recognition model. The update parameter matrix AW can be decomposed into a first low-order decomposition matrix AW e R d×r and a second low-order decomposition matrix BW e R r×k , the first low-order decomposition matrix AW e R d×r and the second low-order decomposition matrix BW e R r×k The ranks of the first low-order decomposition matrix AW and the second low-order decomposition matrix BW are much smaller than the rank of the update parameter matrix, r << min(d, k), and W0 + AW x = W0 x + AW BW x is obtained.

[0289] During training, the parameters in the pre-training model parameter matrix can be frozen, and only the parameters in the first low-order decomposition matrix and the second low-order decomposition matrix are adjusted. The matrix AW is initialized with a random Gaussian, and the matrix BW is initialized to 0.

[0290] FIG. 14 schematically shows a matrix decomposition diagram according to an embodiment of the present disclosure.

[0291] As shown in FIG. 14, the model parameter in the image recognition model is W Q / V / K + AW. The update parameter matrix AW is decomposed into a first low-order decomposition matrix AW and a second low-order decomposition matrix BW. During parameter adjustment, W Q / V / K is frozen, thereby reducing the parameter quantity for parameter adjustment and improving the parameter optimization efficiency.

[0292] According to an embodiment of the present disclosure, the model parameter matrix can be decomposed by using a low-order adaptive model conversion method, thereby reducing the parameter quantity for parameter adjustment and improving the training efficiency.

[0293] Exemplarily, part of the model parameters used for performing task feature enhancement can also be frozen, and part of the model parameters are adjusted to improve the training efficiency.

[0294] For example, during training, the model parameters of the cross-attention layer and the multi-layer perceptron processing layer are frozen, and only the model parameters of the adapter are trained. In this way, the parameter quantity for model parameter adjustment is reduced to improve the training efficiency.

[0295] The above describes reducing model parameters to improve the training efficiency of the image recognition model. The following describes the operations performed on the sample feature map block involved in operation S1310 as shown in FIG. 13.

[0296] In another example, the training method of the image recognition model as shown in FIG. 13 can further include: performing data augmentation on the sample feature map to achieve data augmentation.

[0297] For example, the sample sub-feature map of the sample input image is subjected to data augmentation by Fourier transform to obtain a data transformed sample recognition result. The image recognition model is trained using the data transformed sample recognition result and the sample label.

[0298] Specifically, the sample feature map can be spatially dimensionally divided to obtain a plurality of sample sub-feature maps. I sample sub-feature maps (patches) {P1, P2, …, PI} are obtained, each sample sub-feature map Pi captures a part of the sample feature map. The dimension of each sample sub-feature map is determined by a predefined block size p.

[0299] A Fourier transform can be performed on each sample sub-feature map to obtain a transformed sample sub-feature map.

[0300] Position embedding information can be added to the transformed sample sub-feature map of the augmented feature representation of each sample sub-feature map to preserve the order destroyed in the spatial dimensional division process. The position embedding information of the i-th sample sub-feature map can be represented as Ei, which is a vector aligned with the dimension of the sample sub-feature map. The augmented patch can be obtained by adding the position embedding information and the transformed sample sub-feature map P'i, i.e. SPEi = P'i + Ei, and SPE = {SPE1, SPE2, …, SPEI}. It is worth noting that the position embedding information includes learnable parameters, allowing parameter adjustment during the training of the image recognition model to adapt to different order information.

[0301] Specifically, the sample sub-feature map can be converted into a frequency domain feature by Fourier transform, as shown in equation (15), where the sample sub-feature map can be represented as x[n], 0≤n≤N-1.

[0302] where j represents an imaginary unit, The spectral feature (also referred to as the frequency domain feature) of x[n] at frequency ωk = 2πk / N is represented by X[k], which is periodic within an interval of length N, so only the first N points can be considered.

[0303] The frequency domain feature can be inversely transformed to obtain a transformed sample sub-feature map. See Equation (16) below.

[0304] For X[n], there is a conjugate symmetry, i.e., X[N-k] = X*[k]. The conjugate symmetry can be used to obtain a real-valued discrete signal of the transformed sample sub-feature map based on the frequency domain feature of the sample sub-feature map.

[0305] In another example, the transformed sample sub-feature map can be split in the channel dimension to obtain a plurality of transformed sample feature map blocks. Attention feature extraction is performed on the plurality of transformed sample feature map blocks to obtain a transformed sample channel-enhanced feature map. Task feature enhancement is performed on the transformed sample channel-enhanced feature map to obtain a transformed sample task feature-enhanced feature map. A transformed sample recognition result of an object in the sample input image is obtained based on the transformed sample task feature-enhanced feature map. The image recognition model is trained based on the transformed sample recognition result and a sample label.

[0306] Optionally, an adaptive threshold can also be set. The real-valued discrete signal in X[n] is compared with the adaptive threshold. Data greater than the adaptive threshold is adjusted to the adaptive threshold to obtain the transformed sample sub-feature map.

[0307] Optionally, the real-valued discrete signal in X[n] or the signal adjusted by the adaptive threshold can also be input into an interactive convolution to perform interactive convolution processing to obtain the transformed sample sub-feature map.

[0308] According to embodiments of the present disclosure, feature enhancement is performed on the sample sub-feature map by performing Fourier transform on the sample input image, thereby preventing overfitting problems under a small data set by another data enhancement manner. In addition, adaptive threshold processing can also be performed on the data obtained after Fourier transform to reduce noise in the feature data, thereby reducing noise sensitivity in the training process and improving training efficiency. In addition, an interactive convolution block can also be introduced to perform interactive convolution processing on the transformed sample sub-feature map, so as to use the sample sub-feature map processed by the interactive convolution as the transformed sample sub-feature map, so as to improve the robustness of the image recognition model to different data.

[0309] FIG. 15 schematically shows a flow diagram of a training method of an image recognition model according to an embodiment of the present disclosure.

[0310] As shown in FIG. 15, the training method of the image recognition model can include operations S1510 to operation S1550.

[0311] In operation S1510, a sample image pair is obtained.

[0312] In operation S1520, a network framework of the image recognition model is constructed.

[0313] In operation S1530, an adapter is added in the image recognition model, and a design adapter operation is completed.

[0314] In operation S1540, a low-order adaptive matrix decomposition is performed on a parameter matrix with a large amount of model parameters in the image recognition model, to obtain a low-order model parameter matrix.

[0315] In operation S1550, the image recognition model processed by operations S1520 to S1540 is trained by using the sample image pair obtained in operation S1510.

[0316] In the embodiments of the present disclosure, by improving from different directions such as obtaining a sample image pair, constructing an image recognition model, adding an adapter in the image recognition model, and performing matrix decomposition on model parameters, the training efficiency can be improved, the overfitting problem can be avoided, and the robustness of the image recognition model can be improved.

[0317] The image recognition method and the training method of the image recognition model are explained and described above respectively. The image recognition device and the training device of the image recognition model will be explained and described below.

[0318] The embodiments of the present disclosure provide an image recognition device, which can include an acquisition module, a splitting module, an extraction module, a prompt recognition module, and a segmentation module.

[0319] The acquisition module is configured to acquire an image to be recognized.

[0320] The splitting module is configured to split a feature map of the image to be recognized to obtain a plurality of feature map blocks.

[0321] The extraction module is configured to perform attention feature extraction on the plurality of feature map blocks to obtain a channel-enhanced feature map.

[0322] The prompt recognition module is configured to perform prompt recognition on the image to be recognized to obtain spatial prompt information.

[0323] The segmentation module is configured to perform semantic segmentation on the channel-enhanced feature map based on the spatial prompt information to obtain a recognition map for a target object.

[0324] The embodiments of the present disclosure also provide a training device of an image recognition model, which can include a sample splitting module, a sample extraction module, a sample prompt module, a sample segmentation module, a sample classification module, and a sample training module.

[0325] The sample splitting module is configured to split a sample feature map of a sample input image to obtain a plurality of sample feature map blocks.

[0326] The sample extraction module is configured to perform attention feature extraction on the plurality of sample feature map blocks to obtain a channel-enhanced sample feature map.

[0327] The sample prompt module is configured to perform prompt recognition on the sample input image to obtain sample spatial prompt information.

[0328] The sample segmentation module is configured to perform semantic segmentation on the channel-enhanced sample feature map based on the sample spatial prompt information to obtain a sample recognition map for the target object.

[0329] The sample classification module is configured to perform classification on the sample recognition map to obtain a sample recognition result.

[0330] The sample training module is configured to train the image recognition model based on the sample recognition result and a sample label corresponding to the sample input image. The sample label indicates a category result of the target object.

[0331] Any one or more of the modules according to embodiments of the present disclosure can be implemented in one module. Any one or more of the modules according to embodiments of the present disclosure can be split into multiple modules. Any one or more of the modules according to embodiments of the present disclosure can be implemented at least in part as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application-specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware through integration or packaging of circuits, or in any one of software, hardware, and firmware or in an appropriate combination of any of the foregoing. Alternatively, one or more of the modules according to embodiments of the present disclosure can be implemented at least in part as computer program modules that, when executed, perform corresponding functions.

[0332] For example, the acquisition module, the splitting module, the extraction module, the prompt identification module, and the segmentation module; or any multiple of the sample splitting module, the sample extraction module, the sample prompt module, the sample segmentation module, the sample classification module, and the sample training module can be combined in one module to implement, or any one of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module, the splitting module, the extraction module, the prompt identification module, and the segmentation module; or the sample splitting module, the sample extraction module, the sample prompt module, the sample segmentation module, the sample classification module, and the sample training module can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of integrating or packaging a circuit, etc. hardware or firmware, or in any one of software, hardware, and firmware implementation or in a suitable combination of any of them. At least one of the acquisition module, the splitting module, the extraction module, the prompt identification module, and the segmentation module; or the sample splitting module, the sample extraction module, the sample prompt module, the sample segmentation module, the sample classification module, and the sample training module can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0333] It should be noted that the training device part of the image recognition model in the embodiments of the present disclosure corresponds to the training method part of the image recognition model in the embodiments of the present disclosure, and the description of the training device part of the image recognition model is specifically referred to the training method part of the image recognition model, which will not be repeated here. In addition, the image recognition device part in the embodiments of the present disclosure corresponds to the image recognition method part in the embodiments of the present disclosure, and the description of the image recognition device part is specifically referred to the image recognition method part, which will not be repeated here. FIG. 16 schematically shows a block diagram of an electronic device suitable for implementing the image recognition method and the training method of the image recognition model according to an embodiment of the present disclosure. The electronic device shown in FIG. 16 is only an example and should not limit the functions and use range of the embodiments of the present disclosure.

[0334] As shown in FIG. 16, a computer electronic device 1600 according to an embodiment of the present disclosure includes a processor 1601 that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1602 or a program loaded into a random access memory (RAM) 1603 from a storage section 1609. The processor 1601 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 1601 can also include an on-board memory for cache use. The processor 1601 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present disclosure.

[0335] In the RAM 1603, various programs and data required for the operation of the electronic device 1600 are stored. The processor 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. The processor 1601 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 1602 and / or the RAM 1603. Note that the programs can also be stored in one or more memories other than the ROM 1602 and the RAM 1603. The processor 1601 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0336] According to an embodiment of the present disclosure, the electronic device 1600 can further include an input / output (I / O) interface 1605, which is also connected to the bus 1604. The electronic device 1600 can further include one or more of the following components connected to the input / output (I / O) interface 1605: an input section 1606 including a camera as an input part; an output section 1607 including a display and a speaker as output parts; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, a modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the input / output (I / O) interface 1605 as necessary. A removable medium 1611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1610 as necessary, so that a computer program read out therefrom is installed in the storage section 1608 as necessary.

[0337] According to an embodiment of the present disclosure, the method flow according to the embodiments of the present disclosure can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable storage medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 1609, and / or installed from the detachable medium 1611. When the computer program is executed by the processor 1601, the above-mentioned functions defined in the system of the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0338] The present disclosure also provides a computer-readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, which, when executed, implement the methods according to the embodiments of the present disclosure.

[0339] According to an embodiment of the present disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. For example, it can include but is not limited to portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0340] For example, according to an embodiment of the present disclosure, the computer-readable storage medium can include one or more memories other than the ROM 1602 and / or the RAM 1603 and / or the ROM 1602 and the RAM 1603 described above.

[0341] The embodiments of the present disclosure also include a computer program product including a computer program containing program codes for executing the methods provided by the embodiments of the present disclosure, which, when the computer program product is run on an electronic device, are used for making the electronic device implement the image recognition method and the training method of the image recognition model provided by the embodiments of the present disclosure.

[0342] The computer program, when executed by the processor 1601, performs the above-described functions defined in the system / apparatus of embodiments of the present disclosure. According to embodiments of the present disclosure, the system, apparatus, modules, units, etc. described above can be implemented by way of a computer program.

[0343] In an embodiment, the computer program can be stored in a tangible storage medium, such as an optical, magnetic, or other memory on the device. In another embodiment, the computer program can be transmitted over a network, using a radio wave, infrared signal, or other type of signal. In another embodiment, the computer program can be downloaded from an external source, such as a server, over the Internet or other network. In another embodiment, the computer program can be downloaded from a removable storage medium, such as a CD-ROM, DVD, or memory stick. In another embodiment, the computer program can be downloaded from a server, over the Internet or other network, and then installed on a removable storage medium, such as a CD-ROM, DVD, or memory stick. In another embodiment, the computer program can be downloaded from a server, over the Internet or other network, and then installed on a tangible storage medium, such as an optical, magnetic, or other memory on the device.

[0344] According to embodiments of the present disclosure, program code portions for execution of a computer program implementing embodiments of the present disclosure can be written in any combination of one or more programming languages, including high-level, procedural and / or object oriented programming language, and / or assembly / machine language. Programming languages include, but are not limited to, Java, C++, python, "C", or similar programming languages. Program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote cloud. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0345] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0346] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image recognition method, comprising: obtaining an image to be recognized; splitting a feature map of the image to be recognized to obtain a plurality of feature map blocks; performing attention feature extraction on the plurality of feature map blocks to obtain a channel-enhanced feature map; performing prompt recognition on the image to be recognized to obtain spatial prompt information; and based on the spatial prompt information, performing semantic segmentation on the channel-enhanced feature map to obtain a recognition map for a target object. The performing prompt recognition on the image to be recognized to obtain spatial prompt information comprises:

2. The image recognition method of claim 1, wherein, based on image information of the image to be recognized, determining a plurality of target pixels from the image to be recognized; based on pixel position information of each of the plurality of target pixels, performing region recognition on the image to be recognized to obtain region of interest information; and based on the region of interest information and the pixel position information of each of the plurality of target pixels, obtaining the spatial prompt information. The determining a plurality of target pixels from the image to be recognized based on image information of the image to be recognized comprises:

3. The image recognition method of claim 2, wherein, performing binaryzation processing on the image information of the image to be recognized to obtain a binaryzation image; and based on the binaryzation image, obtaining the plurality of target pixels. The performing region recognition on the image to be recognized based on pixel position information of each of the plurality of target pixels to obtain region of interest information comprises:

4. The image recognition method according to claim 2 or 3, wherein based on the pixel position information of each of the plurality of target pixels, clustering the plurality of target pixels to obtain at least one cluster; and based on image position information of each of the at least one cluster, determining the region of interest information from the image to be recognized. The performing attention feature extraction on the plurality of feature map blocks to obtain a channel-enhanced feature map comprises:

5. The image recognition method of claim 1, wherein, performing parallel attention feature extraction on the plurality of feature map blocks to obtain a plurality of channel-enhanced feature map blocks; and combining the plurality of channel-enhanced feature map blocks to obtain the channel-enhanced feature map. The performing parallel attention feature extraction on the plurality of feature map blocks to obtain the plurality of channel-enhanced feature map blocks comprises:

6. The image recognition method of claim 5, wherein, based on position embedding information, performing global attention feature extraction on the feature map block to obtain the channel-enhanced feature map block, wherein the position embedding information indicates that the feature map block corresponds to an image position in the image to be recognized. The performing parallel attention feature extraction on the plurality of feature map blocks to obtain the plurality of channel-enhanced feature map blocks comprises:

7. The image recognition method of claim 5, wherein, based on position embedding information and segmentation prompt information, performing local attention feature extraction on the feature map block to obtain the channel-enhanced feature map block, wherein the position embedding information indicates that the feature map block corresponds to an image position of the image to be recognized, and the segmentation prompt information indicates an expected feature type. The performing semantic segmentation on the channel-enhanced feature map based on the spatial prompt information to obtain a recognition map for a target object comprises:

8. The image recognition method of claim 1, wherein, ​ performing attention feature extraction on the channel-enhanced feature map based on target prompt information, to obtain an attention feature map, wherein the target prompt information is obtained based on the spatial prompt information, the position embedding information, and segmentation prompt information, and the segmentation prompt information indicates an object type of an expected segmentation; and performing semantic segmentation feature optimization on the attention feature map, to obtain the recognition map.

9. The image recognition method of claim 8, wherein, The performing attention feature extraction on the channel-enhanced feature map based on target prompt information, to obtain an attention feature map, includes: performing cross-attention feature extraction on the channel-enhanced feature map based on the target prompt information, to obtain an initial attention feature map; and performing semantic segmentation feature adaptation on the initial attention feature map, to obtain the attention feature map.

10. The image recognition method according to claim 8 or 9, wherein The performing semantic segmentation feature optimization on the attention feature map, to obtain the recognition map, includes: performing multi-layer perception processing and semantic segmentation feature adaptation on the attention feature map respectively, to obtain a first intermediate feature map and a second intermediate feature map; and performing residual connection on the attention feature map, the first intermediate feature map, and the second intermediate feature map, to obtain the recognition map.

11. The image recognition method of claim 1, wherein, The splitting the feature map of the to-be-recognized image, to obtain a plurality of feature map blocks, includes: performing spatial dimension splitting and channel dimension splitting on the feature map, to obtain a plurality of the feature map blocks.

12. The image recognition method of claim 2, wherein, The combining a plurality of the channel-enhanced feature map blocks includes: performing channel dimension combination and spatial dimension combination on a plurality of the channel-enhanced feature map blocks in sequence, to obtain the channel-enhanced feature map.

13. The image recognition method of claim 1, The obtaining a to-be-recognized image includes: obtaining a plurality of the to-be-recognized images about a target object, wherein the plurality of the to-be-recognized images are collected from different directions of the target object; The method further includes: in a case where a target recognition map characterizes that the target object exists in the target object, determining that the target object exists in the target object, wherein the target recognition map includes a recognition map of at least one of the to-be-recognized images.

14. A training method of an image recognition model, including: splitting a sample feature map of a sample input image, to obtain a plurality of sample feature map blocks; performing attention feature extraction on the plurality of sample feature map blocks, to obtain a channel-enhanced sample feature map; performing prompt recognition on the sample input image, to obtain sample spatial prompt information; performing semantic segmentation on the channel-enhanced sample feature map based on the sample spatial prompt information, to obtain a sample recognition map for a target object; classifying the sample recognition map, to obtain a sample recognition result; and training the image recognition model based on the sample recognition result and a sample label corresponding to the sample input image, wherein the sample label indicates a category result of the target object.

15. The method of claim 14, further including: fusing a target region of a first sample image and a region image other than the target region in a second sample image, to obtain the sample input image; and ​ ​ obtaining the sample label corresponding to the sample input image based on the label of the first sample image, the label of the second sample image and a fusion ratio, wherein the fusion ratio comprises a proportion of the target region in the first sample image; or performing fusion on the target region of the first sample image and the second sample image after contraction to obtain the sample input image; and obtaining the sample label of the sample input image based on the label of the first sample image, the label of the second sample image, a fusion ratio and a contraction ratio, wherein the fusion ratio comprises a proportion of the target region in the first sample image, and the contraction ratio is determined based on a proportion of the target region in the second sample image. 16.An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15. 17.The electronic device of claim 16, further comprising: a camera configured to capture image information about a target item to obtain the image to be recognized; and a sorter configured to sort the target item if the identification chart indicates that the target item exists in the target object. The computer instructions are configured to enable the computer to perform the method of any one of claims 1-15.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, 19.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-15. ​

Images