Image target recognition method and electronic equipment
By using the combination of pre-trained image feature extraction model and convolutional segmentation model, the problem of high resource consumption of image object detection model in the prior art is solved, and efficient real-time object detection is achieved.
Patent Information
- Application Number
- CN202411444706.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-16
- Publication Date
- 2025-01-17
AI Technical Summary
Existing image object detection models require a large amount of labeled data and computing resources to train, and require a long time and high memory consumption during inference, making them unsuitable for object detection scenarios that require real-time response.
The pre-trained image feature extraction model, including an embedding layer and an encoding layer, processes the pending image through the embedding layer to obtain the target input sequence, and obtains the depth feature map through the encoding layer. The depth feature map is processed based on the preset background mask to obtain Regional candidate sets, and finally input the candidate sets into the pre-trained convolutional segmentation model to improve detection speed and accuracy.
It effectively improves the speed and accuracy of image object detection of high-speed shooting instruments, meets the real-time requirement of object detection during use of high-speed shooting instruments, and reduces the inference time and computing resources consumption.
Smart Images
Figure CN119323669A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an image target recognition method and electronic equipment. Background Art
[0002] With the booming development of e-commerce and the rapid growth of the logistics industry, long-distance barcode scanning technology has become increasingly important. In the logistics field, such as express delivery stations and warehousing, quickly and accurately scanning barcodes or QR codes on packages is essential for confirming items, tracking goods and improving work efficiency.
[0003] A camera is a device used to capture and display documents, physical objects or flat objects. It is often used in education, business, demonstrations and various training scenarios, allowing users to easily share, display and record physical documents, books, examples, etc. A camera has a larger scanning range and faster scanning speed, and can accurately read barcodes or QR codes on target objects at a longer distance. Camera image target detection is a challenging task because cameras usually have low resolution, low contrast, high noise, occlusion, deformation and other characteristics due to firmware limitations, making it difficult to identify targets.
[0004] In order to improve the performance and accuracy of image object detection, many deep learning-based methods have emerged in recent years, such as CNN, Swin Transformer, etc. These methods usually require a large amount of labeled data and computing resources for training, and the model takes a long time and consumes a lot of memory during inference, making these models unsuitable for object detection scenarios that require real-time response. Summary of the invention
[0005] The present invention provides an image target recognition method and an electronic device, so as to effectively improve the speed and accuracy of high-speed scanner image target detection, meet the real-time requirements of target detection during the use of the high-speed scanner, and reduce the inference time and the consumption of computing resources.
[0006] According to one aspect of the present invention, there is provided an image object recognition method, comprising:
[0007] Obtaining an image to be processed and a pre-trained image feature extraction model, wherein the image feature extraction model includes at least an embedding layer and an encoding layer, and the encoding layer includes a plurality of attention heads;
[0008] Processing the image to be processed by the embedding layer to obtain a target input sequence corresponding to the image to be processed, and processing the target input sequence by the encoding layer to obtain a depth feature map corresponding to the image to be processed;
[0009] Processing the depth feature map based on a preset background mask to obtain a region candidate set corresponding to the image to be processed, wherein the region candidate set includes an initial background region and an initial target region of the image to be processed;
[0010] The region candidate set is input into a pre-trained convolutional segmentation model to obtain a target object region output by the convolutional segmentation model and corresponding to the image to be processed.
[0011] According to another aspect of the present invention, there is provided an image object recognition device, characterized in that it comprises:
[0012] An image acquisition module, used to acquire an image to be processed and a pre-trained image feature extraction model, wherein the image feature extraction model at least includes an embedding layer and an encoding layer, and the encoding layer includes a plurality of attention heads;
[0013] An image feature extraction module, configured to process the image to be processed through the embedding layer to obtain a target input sequence corresponding to the image to be processed, and to process the target input sequence through the encoding layer to obtain a depth feature map corresponding to the image to be processed;
[0014] A candidate set determination module, used for processing the depth feature map based on a preset background mask to obtain a region candidate set corresponding to the image to be processed, wherein the region candidate set includes an initial background region and an initial target region of the image to be processed;
[0015] The image recognition module is used to input the region candidate set into a pre-trained convolutional segmentation model to obtain a target object region output by the convolutional segmentation model corresponding to the image to be processed.
[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0017] at least one processor;
[0018] and a memory communicatively connected to the at least one processor; wherein,
[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image object recognition method described in any embodiment of the present invention.
[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image target recognition method described in any embodiment of the present invention when executed.
[0021] The technical solution of the embodiment of the present invention, by obtaining the image to be processed and the pre-trained image feature extraction model, wherein the image feature extraction model includes at least an embedding layer and a coding layer, and the coding layer includes multiple attention heads; the image to be processed is processed by the embedding layer to obtain the target input sequence corresponding to the image to be processed, and the target input sequence is processed by the coding layer to obtain the depth feature map corresponding to the image to be processed; the depth feature map is processed based on the preset background mask to obtain the region candidate set corresponding to the image to be processed; the region candidate set is input into the pre-trained convolution segmentation model to obtain the target object region corresponding to the image to be processed output by the convolution segmentation model, which solves the technical problems that the existing image target detection model requires a large amount of labeled data and computing resources to train, and the model requires a long time and high memory consumption during reasoning, making these models unsuitable for target detection scenarios that require real-time response. The present invention can effectively improve the speed and accuracy of high-speed image target detection, meet the real-time requirements of target detection during high-speed use, and reduce the consumption of reasoning time and computing resources. It can effectively improve the speed and accuracy of image target detection, meet the real-time requirements of target detection, and reduce the consumption of reasoning time and computing resources.
[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0024] Figure 1 A flowchart of an image segmentation model training method provided in Embodiment 1 of the present invention;
[0025] Figure 2 A flowchart of an image target recognition method provided by Embodiment 2 of the present invention;
[0026] Figure 3 A schematic diagram of the structure of an image object recognition device provided in Embodiment 3 of the present invention;
[0027] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION
[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0030] Embodiment 1
[0031] Figure 1 This is a flow chart of an image segmentation model training method provided in the first embodiment of the present invention. This embodiment can be applied to target detection in images collected by high-speed cameras. The method can be executed by an image segmentation model training device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:
[0032] S110, obtaining an image to be processed and a pre-trained image feature extraction model.
[0033] The image to be processed may be an image acquired by an image acquisition device, and the image acquisition device may be a high-speed scanner.
[0034] Exemplarily, the image to be processed is an image of a courier item. In order to identify the barcode or QR code in the courier item image, the image can be used as the image to be processed. The image feature extraction model can be a pre-trained Vision Transformer model. Vision Transformer is a deep learning model based on a self-attention mechanism. The model includes at least an embedding layer and an encoding layer. The encoding layer includes multiple attention heads.
[0035] S120, processing the image to be processed through the embedding layer to obtain a target input sequence corresponding to the image to be processed, and processing the target input sequence through the encoding layer to obtain a depth feature map corresponding to the image to be processed.
[0036] Among them, the target input sequence can be a vector sequence corresponding to the image to be processed, and the depth feature map can be a feature map representing the features of the image to be processed.
[0037] Specifically, the image to be processed is processed by the embedding layer to obtain the target input sequence corresponding to the image to be processed, that is, the image to be processed is converted into a vector sequence that can be processed by the target encoding layer. Then, the multi-head attention mechanism of the encoding layer can process the target input sequence to capture the global and local information of the image and obtain the deep feature map corresponding to the image to be processed.
[0038] In the embodiment of the present invention, the embedding layer processes the image to be processed through the following process to obtain a target input sequence corresponding to the image to be processed:
[0039] The image to be processed is divided into a plurality of blocks; for a single block, the block is linearly mapped to a feature space of a preset dimension to obtain a target vector corresponding to the block; a position vector corresponding to the block is determined, and a target input sequence corresponding to the image to be processed is determined based on the position vector and the target vector.
[0040] The target vector may represent content information of the tile, and the position vector may represent position information of the tile.
[0041] Specifically, the image to be processed can be divided into multiple tiles through the embedding layer, and the size of the tiles can be the same. For each tile, the information of the tile is linearly mapped to a feature space of a preset dimension to obtain the target vector corresponding to the tile. In addition to the content information of the tile, the position information of the tile in the image to be processed also needs to be considered, so the position vector corresponding to each tile can be calculated, and the target input sequence of the image to be processed can be constructed according to the target vector and position vector of each tile.
[0042] In an embodiment of the present invention, determining the target input sequence corresponding to the image to be processed based on the position vector and the target vector includes: adding the position vector and the target vector to obtain the vector to be used; multiplying the vector to be used with the weight matrix of the image feature extraction model to obtain the target input sequence corresponding to the processed image.
[0043] Among them, the weight matrix can be W corresponding to the image feature extraction model q , W k , W vThree weight matrices, the target input sequence includes a query vector (Q), a key vector (K) and a value vector (V) corresponding to each of the tiles.
[0044] Specifically, for each tile, the position vector and target vector corresponding to the tile can be added together to combine the content information and position information of the tile to form a vector to be used that contains comprehensive information about the tile. Then, the vector to be used is multiplied by the weight matrix of the image feature extraction model to obtain the query vector, key vector, and value vector corresponding to each tile. The query vectors, key vectors, and value vectors of multiple tiles constitute the target input sequence corresponding to the image to be processed.
[0045] In a preferred implementation, the target input sequence corresponding to the image to be processed can be obtained through the following process:
[0046] Patch Embeddings: Vision Transformer first divides the image into fixed-size patches. For example, for a 224×224 image, it can be divided into 196 16×16 patches.
[0047] Each patch is linearly mapped to a feature space of dimension d to obtain a 1×768 (16×16×3, considering that the image pixels are RGB three-channel values) vector. This vector is the Patch Embeddings, that is, the target vector.
[0048] Positional Encoding (PE): used to map the spatial position of each patch in the patch sequence into a vector, thereby providing position information for the Vision Transformer model. The position information is obtained according to the following formula:
[0049]
[0050]
[0051] pos represents the position index of each patch in the input sequence, which usually starts from 0 and goes to the maximum length of the sequence minus 1. i represents the index in the vector dimension. d represents the total dimension of the input feature vector.
[0052] The position vector of each patch is obtained through the above formula, and the position vector is added to the Patch Embeddings to obtain the vector x to be used. After x passes through the weight matrix, three key vectors are obtained:
[0053] Q=x·W q
[0054] K=x·Wk
[0055] V=x·W v
[0056] In an embodiment of the present invention, the encoding layer processes the target input sequence through the following process to obtain a depth feature map corresponding to the image to be processed, which may include: processing the target input sequence through each attention head corresponding to the encoding layer respectively to obtain an attention output corresponding to each attention head; determining a head attention weight corresponding to each attention head, and obtaining a depth feature map corresponding to the image to be processed based on the attention output and the head attention weight.
[0057] The encoding layer includes multiple attention heads, and each attention head pays different attention to different areas in the processed image.
[0058] Specifically, each attention head of the encoding layer can process the target input sequence separately to obtain the attention output corresponding to each attention head. At the same time, the weight of each attention head is determined, so that the deep feature map of the image to be processed can be calculated based on the attention output of each attention head and the head attention weight.
[0059] In an embodiment of the present invention, the target input sequence is processed separately by each attention head corresponding to the encoding layer to obtain the attention output corresponding to each attention head, including: for a single attention head, based on the dot product of the query vector and the key vector of each tile, determining the attention score corresponding to the single attention head; based on the attention score and the value vector of each tile, determining the attention output corresponding to the single attention head.
[0060] Specifically, for each attention head, the above process can be used to calculate its corresponding attention output. Specifically, the attention score corresponding to a single attention head is obtained based on the dot product of the query vector and the key vector of each tile, and then the attention score is multiplied by the corresponding value vector of each tile to obtain the attention output corresponding to the single attention head.
[0061] Exemplarily, the target input sequence includes Q, K, and V corresponding to each tile. In the first head, it is assumed that there are 196 Query vectors (one for each patch) and 196 Key vectors (also one for each patch). For the query vector of the first patch, you can calculate its dot product with all 196 Key vectors to get a 196-dimensional vector (the first patch attention score). Repeat the above process for each of the 196 patches to get a 196x196 attention score matrix, that is, the attention score of the first head.
[0062] Further, the attention scores can be normalized to convert the attention scores into weights. Usually, the softmax function can be used for normalization to obtain the attention weight matrix. Then, the weighted sum of the Value vectors is performed based on the attention weight matrix. For example, 196 Value vectors (one for each patch) are weighted summed using the attention weight matrix to obtain the attention output of the first head.
[0063] Among them, the calculation formula of the attention weight matrix can be: Attention weight α ij , represents the correlation of the i-th patch to the j-th patch.
[0064] The calculation formula of attention output can be:
[0065] in, is a scaling factor, usually the square root of the dimension of the Key vector, to prevent the dot product result from being too large.
[0066] S i =|{p|A 1i , A 2i …A pi , p = 1, ..., 196}|
[0067] A pi is the attention score of each head. S i It is a set of all patch attention scores, p is the selection of patch, and i is the selection of head.
[0068] It is understandable that the above embodiment describes the calculation process of the attention score, attention weight matrix and attention output of the attention output of a single attention head. However, this embodiment includes multiple attention heads, and the head attention weight corresponding to each attention head can be further calculated.
[0069] In an embodiment of the present invention, determining the head attention weight corresponding to each of the attention heads includes: for a single current attention head, determining the head attention weight corresponding to the current attention head based on the ratio of the attention score corresponding to the current attention head to all attention scores.
[0070] Specifically, we can calculate the importance weight of each head in the deep feature map based on the sparsity of multi-head attention and the number of attention heads as h. This is achieved by considering the contribution of each head to the overall image features, ensuring that more attention is paid to heads with higher information content in subsequent steps:
[0071]
[0072] represents the sum of attention scores of all heads. i represents the head attention weight of the i-th head, and i represents the selection of the head.
[0073] In an embodiment of the present invention, obtaining the depth feature map corresponding to the image to be processed based on the attention output and the head attention weight includes: performing weighted summation based on the attention output and the head attention weight corresponding to each of the attention heads to obtain the depth feature map corresponding to the image to be processed.
[0074] It can be understood that after obtaining the attention output and the head attention weight corresponding to each attention head, a weighted sum can be performed to obtain a depth feature map corresponding to the image to be processed.
[0075] S130: Process the depth feature map based on a preset background mask to obtain a region candidate set corresponding to the image to be processed.
[0076] The candidate region set includes an initial background region and an initial target region of the image to be processed.
[0077] Specifically, the preset background mask can be defined as:
[0078]
[0079] Where T>0; a(H p , H s ) represents the calculation of the cosine similarity of the Key vectors of patch p and seed patch s.
[0080] In an embodiment of the present invention, the depth feature map includes a target weighted attention corresponding to each of the blocks in the image to be processed, and the depth feature map is processed based on a preset background mask to obtain a region candidate set corresponding to the image to be processed, including:
[0081] The image block with the smallest target weighted attention is used as the initial seed block; based on the similarity between each image block and the initial seed block, the background area expansion process is repeatedly performed, so as to obtain a candidate set to be used corresponding to the image to be processed; and the candidate set to be used is filtered to obtain the area candidate set.
[0082] It should be noted that the depth feature map includes the target weighted attention corresponding to each of the blocks in the image to be processed, and the block with the smallest target weighted attention can be used as the initial seed block. Then, the cosine similarity between the initial seed block and each of the remaining blocks is calculated. Specifically, the cosine similarity of the target weighted attention of the two can be calculated. In this way, the blocks are divided into two categories, one of which belongs to the initial background area corresponding to the background part of the image to be processed, and the other belongs to the initial target area corresponding to the target part of the image to be processed. That is, by repeating the process of background area expansion, multiple sets of regions that may contain target areas or background areas are determined after expansion of the initial seed points to form a candidate set to be used. This helps to capture possible different segmentation areas in the image and provide more options for subsequent steps.
[0083] Furthermore, candidate set filtering is performed to filter the generated candidate set to be used, removing areas that are too small or overlap with other areas to obtain a cleaner area candidate set, thereby improving the accuracy of the segmentation result.
[0084] S140: Input the region candidate set into a pre-trained convolutional segmentation model to obtain a target object region output by the convolutional segmentation model that corresponds to the image to be processed.
[0085] Specifically, the region candidate set is input into a pre-trained convolutional segmentation model to segment the target object region in the image to be processed, and the target object region may be a target QR code or barcode region in the image.
[0086] The technical solution of the embodiment of the present invention, by obtaining the image to be processed and the pre-trained image feature extraction model, wherein the image feature extraction model includes at least an embedding layer and a coding layer, and the coding layer includes multiple attention heads; the image to be processed is processed by the embedding layer to obtain the target input sequence corresponding to the image to be processed, and the target input sequence is processed by the coding layer to obtain the depth feature map corresponding to the image to be processed; the depth feature map is processed based on the preset background mask to obtain the region candidate set corresponding to the image to be processed; the region candidate set is input into the pre-trained convolutional segmentation model to obtain the target object region corresponding to the image to be processed output by the convolutional segmentation model, which solves the technical problem that the existing image target detection model requires a large amount of labeled data and computing resources to train, and the model requires a long time and high memory consumption during reasoning, making these models unsuitable for target detection scenarios that require real-time response. The present invention can effectively improve the speed and accuracy of high-speed image target detection, meet the real-time requirements of target detection during high-speed use, and reduce the consumption of reasoning time and computing resources. It can effectively improve the speed and accuracy of image target detection, meet the real-time requirements of target detection, and reduce the consumption of reasoning time and computing resources.
[0087] Embodiment 2
[0088] Figure 2 This is a flowchart of an image target recognition method provided by the second embodiment of the present invention. Based on the above embodiment, this embodiment further illustrates the training process of the convolutional segmentation model. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here. Figure 2 As shown, the method includes:
[0089] S210: Acquire a sample image, and process the sample image based on the image feature extraction model to obtain a sample depth feature map corresponding to the sample image.
[0090] The sample image may be an image acquired in advance, and may be acquired in the same manner as the image to be processed.
[0091] Specifically, a sample image is acquired, and the sample image is processed by an image feature extraction model to obtain a sample depth feature map corresponding to the sample image. The processing process of the sample image by the image feature extraction model may be the same as the processing process of the image to be processed.
[0092] S220: Process the sample depth feature map based on a preset background mask to obtain a sample region candidate set corresponding to the sample image.
[0093] Specifically, the processing method of the sample depth feature map can be the same as the processing method of the aforementioned depth feature map, and what is obtained is the sample region candidate set.
[0094] S230: Training an initial convolutional segmentation model based on the sample region candidate set and a fast bilateral solver to obtain the convolutional segmentation model when the training is completed.
[0095] The initial convolutional segmentation model may be a convolutional segmentation model that has not undergone training iterations, or may be a lightweight 1×1 convolutional layer.
[0096] Specifically, the sample region candidate set is input into the initial convolutional segmentation model, and the lightweight 1×1 convolutional layer is trained using a unique loss function and target and background labels to improve the accuracy of segmentation.
[0097] Loss function optimization: Use the fast bilateral solver to optimize the classifier to make the segmentation edge smoother. By adjusting and optimizing the loss function to reduce the edge between the target object and the background, a more natural and accurate segmentation result is obtained. The loss function consists of two parts: the binary cross entropy L1 of the output of the segmentation layer and the value calculated by the fast bilateral solver with the background mask:
[0098]
[0099] in, is the output of the convolutional layer segmentation layer. yes Results after passing through the fast bilateral solver.
[0100] The cross entropy L2 between the split layer output and the output calculated by the fast bilateral solver:
[0101]
[0102] is the output Results after passing through the fast bilateral solver.
[0103] The total loss function L=L1+λL2.
[0104] S240: Obtain an image to be processed and a pre-trained image feature extraction model.
[0105] S250, processing the image to be processed through the embedding layer to obtain a target input sequence corresponding to the image to be processed, and processing the target input sequence through the encoding layer to obtain a depth feature map corresponding to the image to be processed.
[0106] S260: Process the depth feature map based on a preset background mask to obtain a region candidate set corresponding to the image to be processed.
[0107] S270: Input the region candidate set into a pre-trained convolutional segmentation model to obtain a target object region output by the convolutional segmentation model that corresponds to the image to be processed.
[0108] With the continuous development of image recognition, although the recognition accuracy is constantly improving, most of the current methods are still significantly behind the strong supervision method. In order to reduce the workload and subjective bias of manual annotation, and to adapt to the use scenarios of high-speed scanners, the present invention uses unsupervised learning methods to apply high-speed scanners to quickly locate target images and realize long-distance code scanning. The present invention can effectively improve the speed and accuracy of high-speed scanner image target detection, meet the real-time requirements of target detection during the use of high-speed scanners, and reduce the consumption of reasoning time and computing resources.
[0109] In an embodiment of the present invention, an unsupervised lightweight model is used to perform image target detection, and a fast bilateral solver is used to smooth the target edge. Unsupervised learning refers to allowing the machine to self-learn and discover the intrinsic structure of the data without any labels or instructions. In unsupervised learning, the algorithm finds patterns in the data by itself, rather than predicting outputs from pre-given labels, and can segment target areas from the data without labels. The fast bilateral solver assumes that image smooth areas must be closely related internally, and believes that there must be consistency or gradual changes when calculating certain visual tasks in these areas. It is 10 to 1000 times faster than similar algorithms and can greatly improve various visual tasks. The present invention is based on unsupervised learning and a fast bilateral solver to propose an image target recognition method.
[0110] Embodiment 3
[0111] Figure 3 This is a schematic diagram of the structure of an image target recognition device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:
[0112] An image acquisition module 310, used to acquire an image to be processed and a pre-trained image feature extraction model, wherein the image feature extraction model at least includes an embedding layer and an encoding layer, and the encoding layer includes multiple attention heads;
[0113] An image feature extraction module 320 is used to process the image to be processed through the embedding layer to obtain a target input sequence corresponding to the image to be processed, and to process the target input sequence through the encoding layer to obtain a depth feature map corresponding to the image to be processed;
[0114] A candidate set determination module 330 is used to process the depth feature map based on a preset background mask to obtain a region candidate set corresponding to the image to be processed, wherein the region candidate set includes an initial background region and an initial target region of the image to be processed;
[0115] The image recognition module 340 is used to input the region candidate set into a pre-trained convolutional segmentation model to obtain a target object region output by the convolutional segmentation model that corresponds to the image to be processed.
[0116] Optionally, the image feature extraction module 320 includes:
[0117] The to-be-processed image division submodule is used to divide the to-be-processed image into a plurality of image blocks;
[0118] A target vector determination submodule, for linearly mapping a single image block to a feature space of a preset dimension to obtain a target vector corresponding to the image block;
[0119] The target input sequence determination submodule is used to determine the position vector corresponding to the image block, and determine the target input sequence corresponding to the image to be processed based on the position vector and the target vector.
[0120] Optionally, the target input sequence determination submodule includes:
[0121] a to-be-used vector determination unit, configured to add the position vector and the target vector to obtain a to-be-used vector;
[0122] A target input sequence determination unit, configured to multiply the vector to be used by a weight matrix of the image feature extraction model to obtain a target input sequence corresponding to the processed image;
[0123] The target input sequence includes a query vector, a key vector and a value vector corresponding to each of the tiles.
[0124] Optionally, the image feature extraction module 320 includes:
[0125] An attention output calculation submodule, used to process the target input sequence respectively through each attention head corresponding to the encoding layer to obtain an attention output corresponding to each attention head;
[0126] The depth feature map determination submodule is used to determine the head attention weight corresponding to each of the attention heads, and obtain the depth feature map corresponding to the image to be processed based on the attention output and the head attention weight.
[0127] Optionally, the attention output calculation submodule includes:
[0128] an attention score calculation unit, configured to determine, for a single attention head, an attention score corresponding to the single attention head based on a dot product of a query vector and a key vector of each of the tiles;
[0129] An attention output calculation unit is used to determine the attention output corresponding to the single attention head based on the attention score and the value vector of each of the tiles.
[0130] Optionally, the depth feature map determination submodule includes:
[0131] A head attention weight determination unit is used to determine, for a single current attention head, the head attention weight corresponding to the current attention head based on the ratio of the attention score corresponding to the current attention head to all attention scores.
[0132] Optionally, the depth feature map determination submodule includes:
[0133] A depth feature map determining unit is used to obtain a depth feature map corresponding to the image to be processed by performing weighted summation based on the attention output corresponding to each of the attention heads and the head attention weight.
[0134] Optionally, the depth feature map includes a target weighted attention corresponding to each of the blocks in the image to be processed, and the candidate set determination module 330 is specifically used to:
[0135] Taking the image block with the smallest target weighted attention as the initial seed block;
[0136] Based on the similarity between each of the image blocks and the initial seed block, and the preset background mask function, the process of repeatedly performing background area expansion is performed to obtain a candidate set to be used corresponding to the image to be processed;
[0137] The candidate set to be used is filtered to obtain the region candidate set.
[0138] Optionally, the device further includes a convolutional segmentation model training module, specifically used for:
[0139] Acquire a sample image, and process the sample image based on the image feature extraction model to obtain a sample depth feature map corresponding to the sample image;
[0140] Processing the sample depth feature map based on a preset background mask to obtain a sample region candidate set corresponding to the sample image;
[0141] The initial convolutional segmentation model is trained based on the sample region candidate set and the fast bilateral solver to obtain the convolutional segmentation model when the training is completed.
[0142] The image object recognition device provided in the embodiment of the present invention can execute the image object recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0143] Embodiment 4
[0144] Figure 4 A schematic diagram of the structure of an electronic device provided for Embodiment 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0145] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0146] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0147] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image target recognition method.
[0148] In some embodiments, the image object recognition method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image object recognition method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image object recognition method in any other appropriate manner (e.g., by means of firmware).
[0149] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0150] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0151] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0152] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0153] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0154] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0155] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0156] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for image target recognition, characterized in that: include: Obtaining an image to be processed and a pre-trained image feature extraction model, wherein the image feature extraction model includes at least an embedding layer and an encoding layer, and the encoding layer includes a plurality of attention heads; Processing the image to be processed by the embedding layer to obtain a target input sequence corresponding to the image to be processed, and processing the target input sequence by the encoding layer to obtain a depth feature map corresponding to the image to be processed; Processing the depth feature map based on a preset background mask to obtain a region candidate set corresponding to the image to be processed, wherein the region candidate set includes an initial background region and an initial target region of the image to be processed; The region candidate set is input into a pre-trained convolutional segmentation model to obtain a target object region output by the convolutional segmentation model and corresponding to the image to be processed.
2. The method according to claim 1, characterized in that The embedding layer processes the image to be processed through the following process to obtain a target input sequence corresponding to the image to be processed: Dividing the image to be processed into a plurality of blocks; For a single image block, linearly map the image block to a feature space of a preset dimension to obtain a target vector corresponding to the image block; A position vector corresponding to the image block is determined, and a target input sequence corresponding to the image to be processed is determined based on the position vector and the target vector.
3. The method according to claim 2, characterized in that The step of determining a target input sequence corresponding to the image to be processed based on the position vector and the target vector comprises: Adding the position vector and the target vector to obtain a vector to be used; Multiplying the vector to be used by the weight matrix of the image feature extraction model to obtain a target input sequence corresponding to the processed image; The target input sequence includes a query vector, a key vector and a value vector corresponding to each of the tiles.
4. The method according to claim 1, characterized in that: The encoding layer processes the target input sequence through the following process to obtain a depth feature map corresponding to the image to be processed: Processing the target input sequence respectively through each attention head corresponding to the encoding layer to obtain an attention output corresponding to each attention head; Determine a head attention weight corresponding to each of the attention heads, and obtain a depth feature map corresponding to the image to be processed based on the attention output and the head attention weight.
5. The method according to claim 4, characterized in that The target input sequence is processed respectively by each attention head corresponding to the encoding layer to obtain an attention output corresponding to each attention head, including: For a single attention head, determining an attention score corresponding to the single attention head based on a dot product of a query vector and a key vector for each tile; Based on the attention score and the value vector of each of the tiles, an attention output corresponding to the single attention head is determined.
6. The method according to claim 5, characterized in that The determining the head attention weight corresponding to each of the attention heads comprises: For a single current attention head, the head attention weight corresponding to the current attention head is determined based on the ratio of the attention score corresponding to the current attention head to all attention scores.
7. The method according to claim 4, characterized in that The obtaining a depth feature map corresponding to the image to be processed based on the attention output and the head attention weight includes: Based on the attention output corresponding to each of the attention heads and the head attention weight, a weighted sum is performed to obtain a depth feature map corresponding to the image to be processed.
8. The method according to claim 2, characterized in that: The depth feature map includes the target weighted attention corresponding to each of the blocks in the image to be processed, and the depth feature map is processed based on a preset background mask to obtain a region candidate set corresponding to the image to be processed, including: Taking the image block with the smallest target weighted attention as the initial seed block; Based on the similarity between each of the image blocks and the initial seed block, and the preset background mask function, the process of repeatedly performing background area expansion is performed to obtain a candidate set to be used corresponding to the image to be processed; The candidate set to be used is filtered to obtain the region candidate set.
9. The method according to claim 1, characterized in that: The convolutional segmentation model is trained through the following process: Acquire a sample image, and process the sample image based on the image feature extraction model to obtain a sample depth feature map corresponding to the sample image; Processing the sample depth feature map based on a preset background mask to obtain a sample region candidate set corresponding to the sample image; The initial convolutional segmentation model is trained based on the sample region candidate set and the fast bilateral solver to obtain the convolutional segmentation model when the training is completed.
10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the image object recognition method according to any one of claims 1 to 9.